Video signal processing method and apparatus using multiple transform kernel

ABSTRACT

A video signal processing method may comprise the steps of: confirming a prediction mode applied to a current coding unit; confirming whether a plurality of preset conditions are satisfied on the basis of at least one of the prediction mode of the current coding unit and a size of the current coding unit; parsing a first syntax element indicating a transform kernel applied to a transform unit included in the current coding unit when the plurality of preset conditions are satisfied; determining a transform kernel applied to horizontal and vertical directions of the current transform unit on the basis of the first syntax element; and generating a residual signal of the current transform unit by performing an inverse transform on the current transform unit using the determined transform kernel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending PCT InternationalApplication No. PCT/KR2020/005402, which was filed on Apr. 23, 2020, andwhich claims priority under 35 U.S.C 119(a) to Korean Patent ApplicationNo. 10-2019-0047006 filed with the Korean Intellectual Property Officeon Apr. 23, 2019, and Korean Patent Application No. 10-2019-0116474filed with the Korean Intellectual Property Office on Sep. 21, 2019. Thedisclosures of the above patent applications are incorporated herein byreference in their entirety.

TECHNICAL FIELD

The present disclosure relates to a method and an apparatus forprocessing a video signal and, more particularly, to a video signalprocessing method and apparatus for encoding and decoding a videosignal.

BACKGROUND ART

Compression coding refers to a series of signal processing techniquesfor transmitting digitized information through a communication line orstoring information in a form suitable for a storage medium. An objectof compression encoding includes objects such as voice, video, and text,and in particular, a technique for performing compression encoding on animage is referred to as video compression. Compression coding for avideo signal is performed by removing excess information inconsideration of spatial correlation, temporal correlation, andstochastic correlation. However, with the recent development of variousmedia and data transmission media, a more efficient video signalprocessing method and apparatus are required.

DISCLOSURE OF THE INVENTION Technical Problem

An aspect of the present disclosure is to increase coding efficiency ofa video signal. Further, another aspect of the present disclosure is toincrease signaling efficiency related to a motion information set of acurrent block.

In particular, an object of the present invention is to provide a methodfor efficiently determining a transform kernel for an intra block copy(IBC) mode. In addition, an object of the present invention is toprovide a method for efficiently determining a transform kernel for animplicitly split transform unit on the basis of a luma maximum transformsize.

Technical Solution

In order to solve the problems described above, the present inventionprovides the following video signal processing apparatus and videosignal processing method.

According to an embodiment of the present invention, a method forprocessing a video signal further comprising: checking a prediction modeapplied to a current coding unit; checking whether a plurality of presetconditions are satisfied on the basis of at least one of a predictionmode of the current coding unit or a size of the current coding unit;parsing when the plurality of preset conditions are satisfied, a firstsyntax element indicating a transform kernel applied to a transform unitincluded in the current coding unit; determining a transform kernelapplied to horizontal and vertical directions of a current transformunit on the basis of the first syntax element; and generating a residualsignal of the current transform unit by performing an inverse transformon the current transform unit using the determined transform kernel.

As an embodiment, when the plurality of preset conditions are notsatisfied, the first syntax element may be inferred as 0.

As an embodiment, when the first syntax element is 0, the transformkernel applied to the horizontal and vertical directions of the currenttransform unit may be determined as a discrete cosine transform type-2(DCT-2)-based transform kernel.

As an embodiment, a method for processing a video signal furthercomprising: obtaining, in a case of an intra mode, a second syntaxelement indicating whether or not a multiple transform selection (MTS)mode is able to be used in a current sequence, the MTS mode representinga mode in which transform is performed using a transform kernel selectedfrom among a plurality of kernels; and obtaining, in a case of an intermode, a third syntax element indicating whether or not the MTS mode isable to be used in the current sequence may be further included.

As an embodiment, the plurality of preset conditions may include whetheror not the prediction mode of the current coding unit is the intra modeand the second syntax element is true, or the prediction mode of thecurrent coding unit is the inter mode and the third syntax element istrue.

As an embodiment, when the prediction mode of the current coding unit isan intra block copy mode, the plurality of preset conditions may bedetermined as being not satisfied regardless of the second syntaxelement and the third syntax element.

As an embodiment, the plurality of preset conditions may include whetheror not a width and height of the current coding unit are less than orequal to 32.

As an embodiment, a method for processing a video signal furthercomprising: receiving information related to a luma maximum transformsize of a current sequence may be further included, and the currentcoding unit, when at least one of the width and height of the currentcoding unit is greater than the luma maximum transform size, may besplit into a plurality of transform units.

As an embodiment, the luma maximum transform size may be 32 or 64.

As an embodiment, when the current coding unit includes the plurality oftransform units split on the basis of the luma maximum transform size,the plurality of preset conditions may be determined as being notsatisfied.

According to an embodiment of the present invention, a video signalprocessing apparatus includes a processor, the processor is configuredto check a prediction mode applied to a current coding unit, checkwhether a plurality of preset conditions are satisfied on the basis ofat least one of a prediction mode of the current coding unit or a sizeof the current coding unit, parse, when the plurality of presetconditions are satisfied, a first syntax element indicating a transformkernel applied to a transform unit included in the current coding unit,determine a transform kernel applied to horizontal and verticaldirections of a current transform unit on the basis of the first syntaxelement, and generate a residual signal of the current transform unit byperforming an inverse transform on the current transform unit using thedetermined transform kernel.

As an embodiment, when the plurality of preset conditions are notsatisfied, the first syntax element may be inferred as 0.

As an embodiment, when the first syntax element is 0, the transformkernel applied to the horizontal and vertical directions of the currenttransform unit may be determined as a discrete cosine transform type-2(DCT-2)-based transform kernel.

As an embodiment, the processor may be configured to obtain, in a caseof an intra mode, a second syntax element indicating whether or not amultiple transform selection (MTS) mode is able to be used in a currentsequence, the MTS mode representing a mode in which transform isperformed using a transform kernel selected from among a plurality ofkernels, and obtain, in a case of an inter mode, a third syntax elementindicating whether or not the MTS mode is able to be used in the currentsequence.

As an embodiment, the plurality of preset conditions may include whetheror not the prediction mode of the current coding unit is the intra modeand the second syntax element is true, or the prediction mode of thecurrent coding unit is the inter mode and the third syntax element istrue.

As an embodiment, when the prediction mode of the current coding unit isan intra block copy mode, the plurality of preset conditions may bedetermined as being not satisfied regardless of the second syntaxelement and the third syntax element.

As an embodiment, the plurality of preset conditions may include whetheror not a width and height of the current coding unit are less than orequal to 32.

As an embodiment, the processor may be configured to receive informationrelated to a luma maximum transform size of a current sequence, and thecurrent coding unit, when at least one of the width and height of thecurrent coding unit is greater than the luma maximum transform size, maybe split into a plurality of transform units.

As an embodiment, the luma maximum transform size may be 32 or 64.

As an embodiment, when the current coding unit includes the plurality oftransform units split on the basis of the luma maximum transform size,the plurality of preset conditions may be determined as being notsatisfied.

According to an embodiment of the present invention, a method forprocessing a video signal comprising: checking a prediction mode appliedto a current coding unit; checking whether a plurality of presetconditions are satisfied on the basis of at least one of a predictionmode of the current coding unit or a size of the current coding unit;encoding when the plurality of preset conditions are satisfied, a firstsyntax element indicating a transform kernel applied to a transform unitincluded in the current coding unit; determining a transform kernelapplied to horizontal and vertical directions of a current transformunit on the basis of the first syntax element; and generating atransform block of the current transform unit by performing a transformon a residual signal of the current transform unit using the determinedtransform kernel.

According to an embodiment of the present invention, a non-transitorycomputer-readable medium that stores a computer-executable componentconfigured to be executed on one or more processors of a computingdevice, the computer-executable component being configured to check aprediction mode applied to a current coding unit, check whether aplurality of preset conditions are satisfied on the basis of at leastone of a prediction mode of the current coding unit or a size of thecurrent coding unit, parse, when the plurality of preset conditions aresatisfied, a first syntax element indicating a transform kernel appliedto a transform unit included in the current coding unit, determine atransform kernel applied to horizontal and vertical directions of acurrent transform unit on the basis of the first syntax element, andgenerate a residual signal of the current transform unit by performingan inverse transform on the current transform unit using the determinedtransform kernel.

Advantageous Effects

According to an embodiment of the present invention, coding efficiencyof a video signal can be improved. In addition, according to anembodiment of the present invention, a transform kernel suitable for thecurrent transform block can be selected.

According to an embodiment of the present invention, by efficientlydetermining a transform kernel for an intra block copy (IBC) mode, it ispossible to increase compression efficiency and save signaling bits fordetermining a transform kernel. In addition, according to an embodimentof the present invention, by efficiently determining a transform kernelfor a transform unit implicitly split on the basis of the luma maximumtransform size, it is possible to increase compression efficiency andsave signaling bits for determining a transform kernel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of a video signal encoding apparatusaccording to an embodiment of the present invention.

FIG. 2 is a schematic block diagram of a video signal decoding apparatusaccording to an embodiment of the present invention.

FIG. 3 shows an embodiment in which a coding tree unit is divided intocoding units in a picture.

FIG. 4 shows an embodiment of a method for signaling a division of aquad tree and a multi-type tree.

FIGS. 5 and 6 illustrate an intra-prediction method in more detailaccording to an embodiment of the present disclosure.

FIG. 7 shows an inter prediction method according to an embodiment ofthe present invention.

FIG. 8 is a diagram specifically illustrating a method for transforminga residual signal by an encoder.

FIG. 9 is a diagram specifically illustrating a method for obtaining aresidual signal by inverse transforming a transform coefficient by anencoder and a decoder.

FIG. 10 is a diagram representing a basis function corresponding to eachtransform type.

FIG. 11 is a diagram representing a size of a signal for each index ofeach of transform types DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII.

FIG. 12 is a diagram representing a size of a signal for each index foreach of transform types DST-IV, DCT-IV, DST-VII, and DCT-VIII.

FIG. 13 is a diagram representing a method of configuring a transformcandidate set according to a prediction mode of a current block.

FIG. 14 represents a horizontal transform kernel and a verticaltransform kernel determined according to a set index.

FIG. 15 is a diagram representing a method of obtainingtransform-related information according to an embodiment of the presentinvention.

FIG. 16 is a diagram for describing a method of selecting a transformkernel on the basis of a shape of a block according to an embodiment ofthe present invention.

FIG. 17 is a diagram illustrating a method of signaling and determininga transform kernel according to an embodiment of the present invention.

FIG. 18 represents a syntax structure for obtaining information relatedto a multiple transform selection and transform skip according to anembodiment of the present invention, and FIG. 19 is a diagramillustrating information related to a multiple transform selection andtransform skip according to an embodiment of the present invention.

FIG. 20 is a diagram for describing a sub-block transform (SBT) methodaccording to an embodiment of the present invention.

FIG. 21 is a diagram illustrating a method of splitting a block in anintra sub-partitions encoding mode according to an embodiment of thepresent invention.

FIG. 22 is a diagram illustrating a method of selecting, if the intrasub-partitions mode according to an embodiment of the present inventionis applied, a transform kernel applied to a split transform unit.

FIG. 23 shows an example of a transform unit syntax structure and anexample of a method of determining a transform type according to anembodiment to which the present invention is applied.

FIG. 24 is a diagram illustrating a structure of a high level syntax anda transform unit syntax according to an embodiment to which the presentinvention is applied.

FIG. 25 is a diagram illustrating a method of selecting, if IBCprediction is applied, a transform method, as an embodiment to which thepresent invention is applied.

FIG. 26 is a diagram showing an example of a transform tree syntaxstructure according to an embodiment of the present invention.

FIG. 27 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention.

FIG. 28 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention.

FIG. 29 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention.

FIG. 30 is a flowchart representing a video signal processing methodaccording to an embodiment of the present invention.

MODE FOR CARRYING OUT THE INVENTION

Terms used in this specification may be currently widely used generalterms in consideration of functions in the present invention but mayvary according to the intents of those skilled in the art, customs, orthe advent of new technology. Additionally, in certain cases, there maybe terms the applicant selects arbitrarily and, in this case, theirmeanings are described in a corresponding description part of thepresent invention. Accordingly, terms used in this specification shouldbe interpreted based on the substantial meanings of the terms andcontents over the whole specification.

In this specification, some terms may be interpreted as follows. Codingmay be interpreted as encoding or decoding in some cases. In the presentspecification, an apparatus for generating a video signal bitstream byperforming encoding (coding) of a video signal is referred to as anencoding apparatus or an encoder, and an apparatus that performsdecoding (decoding) of a video signal bitstream to reconstruct a videosignal is referred to as a decoding apparatus or decoder. In addition,in this specification, the video signal processing apparatus is used asa term of a concept including both an encoder and a decoder. Informationis a term including all values, parameters, coefficients, elements, etc.In some cases, the meaning is interpreted differently, so the presentinvention is not limited thereto. ‘Unit’ is used as a meaning to referto a basic unit of image processing or a specific position of a picture,and refers to an image region including both a luma component and achroma component. In addition, ‘block’ refers to an image regionincluding a specific component among luma components and chromacomponents (i.e., Cb and Cr). However, depending on the embodiment,terms such as ‘unit’, ‘block’, ‘partition’ and ‘region’ may be usedinterchangeably. In addition, in this specification, a unit may be usedas a concept including all of a coding unit, a prediction unit, and atransform unit. The picture indicates a field or frame, and according toan embodiment, the terms may be used interchangeably.

FIG. 1 is a schematic block diagram of a video signal encoding apparatusaccording to an embodiment of the present invention. Referring to FIG.1, the encoding apparatus 100 of the present invention includes atransformation unit 110, a quantization unit 115, an inversequantization unit 120, an inverse transformation unit 125, a filteringunit 130, a prediction unit 150, and an entropy coding unit 160.

The transformation unit 110 obtains a value of a transform coefficientby transforming a residual signal, which is a difference between theinputted video signal and the predicted signal generated by theprediction unit 150. For example, a Discrete Cosine Transform (DCT), aDiscrete Sine Transform (DST), or a Wavelet Transform may be used. TheDCT and DST perform transformation by splitting the input picture signalinto blocks. In the transformation, coding efficiency may vary accordingto the distribution and characteristics of values in the transformationregion. The quantization unit 115 quantizes the value of the transformcoefficient value outputted from the transformation unit 110.

In order to improve coding efficiency, instead of coding the picturesignal as it is, a method of predicting a picture using a region alreadycoded through the prediction unit 150 and obtaining a reconstructedpicture by adding a residual value between the original picture and thepredicted picture to the predicted picture is used. In order to preventmismatches in the encoder and decoder, information that may be used inthe decoder should be used when performing prediction in the encoder.For this, the encoder performs a process of reconstructing the encodedcurrent block again. The inverse quantization unit 120 inverse-quantizesthe value of the transform coefficient, and the inverse transformationunit 125 reconstructs the residual value using the inverse quantizedtransform coefficient value. Meanwhile, the filtering unit 130 performsfiltering operations to improve the quality of the reconstructed pictureand to improve the coding efficiency. For example, a deblocking filter,a sample adaptive offset (SAO), and an adaptive loop filter may beincluded. The filtered picture is outputted or stored in a decodedpicture buffer (DPB) 156 for use as a reference picture.

In order to improve coding efficiency, a picture signal is not coded asit is, but a method of predicting a picture via the prediction unit 150by using a region that has been already coded, and adding, to thepredicted picture, a residual value between an original picture and thepredicted picture, thereby obtaining a reconstructed picture, is used.The intra prediction unit 152 performs intra prediction within a currentpicture, and the inter prediction unit 154 predicts the current pictureby using a reference picture stored in the decoding picture buffer 156.The intra prediction unit 152 performs intra prediction fromreconstructed regions in the current picture, and transfers intra codinginformation to the entropy coding unit 160. The inter prediction unit154 may include a motion estimation unit 154 a and a motion compensationunit 154 b. The motion estimation unit 154 a obtains a motion vectorvalue of the current region by referring to a specific reconstructedregion. The motion estimation unit 154 a transfers location information(reference frame, motion vector, etc.) of the reference region to theentropy coding unit 160 so as to enable the location information to beincluded in a bitstream. The motion compensation unit 154 b performsinter motion compensation by using the motion vector value transferredfrom the motion estimation unit 154 a.

The prediction unit 150 includes an intra prediction unit 152 and aninter prediction unit 154. The intra prediction unit 152 performs intraprediction in the current picture, and the inter prediction unit 154performs inter prediction to predict the current picture by using thereference picture stored in the DBP 156. The intra prediction unit 152performs intra prediction from reconstructed samples in the currentpicture, and transfers intra encoding information to the entropy codingunit 160. The intra encoding information may include at least one of anintra prediction mode, a most probable mode (MPM) flag, and an MPMindex. The intra encoding information may include information on areference sample. The inter prediction unit 154 may include the motionestimation unit 154 a and the motion compensation unit 154 b. The motionestimation unit 154 a obtains a motion vector value of the currentregion by referring to a specific region of the reconstructed referencepicture. The motion estimation unit 154 a transfers a motion informationset (reference picture index, motion vector information, etc.) for thereference region to the entropy coding unit 160. The motion compensationunit 154 b performs motion compensation by using the motion vector valuetransferred from the motion estimation unit 154 a. The inter predictionunit 154 transfers inter encoding information including motioninformation on the reference region to the entropy coding unit 160.

According to an additional embodiment, the prediction unit 150 mayinclude an intra-block copy (BC) prediction unit (not shown). Theintra-BC prediction unit performs intra-BC prediction based onreconstructed samples in the current picture, and transmits intra-BCencoding information to the entropy coding unit 160. The intra-BCprediction unit obtains a block vector value indicating a reference areaused for predicting a current area with reference to a specific area inthe current picture. The intra-BC prediction unit may perform intra-BCprediction using the obtained block vector value. The intra-BCprediction unit transmits intra-BC encoding information to the entropycoding unit 160. The intra-BC encoding information may include blockvector information.

When the picture prediction described above is performed, thetransformation unit 110 transforms a residual value between the originalpicture and the predicted picture to obtain a transform coefficientvalue. In this case, the transformation may be performed in a specificblock unit within a picture, and the size of a specific block may bevaried within a preset range. The quantization unit 115 quantizes thetransform coefficient value generated in the transformation unit 110 andtransmits it to the entropy coding unit 160.

The entropy coding unit 160 entropy-codes information indicating aquantized transform coefficient, intra-encoding information,inter-encoding information, and the like to generate a video signalbitstream. In the entropy coding unit 160, a variable length coding(VLC) scheme, an arithmetic coding scheme, etc. may be used. Thevariable length coding (VLC) scheme includes transforming input symbolsinto consecutive codewords, and a length of a codeword may be variable.For example, frequently occurring symbols are represented by a shortcodeword, and infrequently occurring symbols are represented by a longcodeword. A context-based adaptive variable length coding (CAVLC) schememay be used as a variable length coding scheme. Arithmetic coding maytransform continuous data symbols into a single prime number, whereinarithmetic coding may obtain an optimal bit required for representingeach symbol. A context-based adaptive binary arithmetic code (CABAC) maybe used as arithmetic coding. For example, the entropy coding unit 160may binarize information indicating a quantized transform coefficient.The entropy coding unit 160 may generate a bitstream byarithmetic-coding the binary information.

The generated bitstream is encapsulated using a network abstractionlayer (NAL) unit as a basic unit. The NAL unit includes an integernumber of coded coding tree units. In order to decode a bitstream in avideo decoder, first, the bitstream must be separated in NAL units, andthen each separated NAL unit must be decoded. Meanwhile, informationnecessary for decoding a video signal bitstream may be transmittedthrough an upper level set of Raw Byte Sequence Payload (RBSP) such asPicture Parameter Set (PPS), Sequence Parameter Set (SPS), VideoParameter Set (VPS), and the like.

Meanwhile, the block diagram of FIG. 1 shows an encoding apparatus 100according to an embodiment of the present invention, and separatelydisplayed blocks logically distinguish and show the elements of theencoding apparatus 100. Accordingly, the elements of the above-describedencoding apparatus 100 may be mounted as one chip or as a plurality ofchips depending on the design of the device. According to an embodiment,the operation of each element of the above-described encoding apparatus100 may be performed by a processor (not shown).

FIG. 2 is a schematic block diagram of a video signal decoding apparatus200 according to an embodiment of the present invention. Referring toFIG. 2, the decoding apparatus 200 of the present invention includes anentropy decoding unit 210, an inverse quantization unit 220, an inversetransformation unit 225, a filtering unit 230, and a prediction unit250.

The entropy decoding unit 210 entropy-decodes a video signal bitstreamto extract transform coefficient information, intra encodinginformation, inter encoding information, and the like for each region.For example, the entropy decoding unit 210 may obtain a binarizationcode for transform coefficient information of a specific region from thevideo signal bitstream. The entropy decoding unit 210 obtains aquantized transform coefficient by inverse-binarizing a binary code. Theinverse quantization unit 220 inverse-quantizes the quantized transformcoefficient, and the inverse transformation unit 225 reconstructs aresidual value by using the inverse-quantized transform coefficient. Thevideo signal processing device 200 reconstructs an original pixel valueby summing the residual value obtained by the inverse transformationunit 225 with a prediction value obtained by the prediction unit 250.

Meanwhile, the filtering unit 230 performs filtering on a picture toimprove image quality. This may include a deblocking filter for reducingblock distortion and/or an adaptive loop filter for removing distortionof the entire picture. The filtered picture is outputted or stored inthe DPB 256 for use as a reference picture for the next picture.

The prediction unit 250 includes an intra prediction unit 252 and aninter prediction unit 254. The prediction unit 250 generates aprediction picture by using the encoding type decoded through theentropy decoding unit 210 described above, transform coefficients foreach region, and intra/inter encoding information. In order toreconstruct a current block in which decoding is performed, a decodedregion of the current picture or other pictures including the currentblock may be used. In a reconstruction, only a current picture, that is,a picture (or, tile/slice) that performs intra prediction or intra BCprediction, is called an intra picture or an I picture (or, tile/slice),and a picture (or, tile/slice) that may perform all of intra prediction,inter prediction, and intra BC prediction is called an inter picture(or, tile/slice). In order to predict sample values of each block amonginter pictures (or, tiles/slices), a picture (or, tile/slice) using upto one motion vector and a reference picture index is called apredictive picture or P picture (or, tile/slice), and a picture (ortile/slice) using up to two motion vectors and a reference picture indexis called a bi-predictive picture or a B picture (or tile/slice). Inother words, the P picture (or, tile/slice) uses up to one motioninformation set to predict each block, and the B picture (or,tile/slice) uses up to two motion information sets to predict eachblock. Here, the motion information set includes one or more motionvectors and one reference picture index.

The intra prediction unit 252 generates a prediction block using theintra encoding information and reconstructed samples in the currentpicture. As described above, the intra encoding information may includeat least one of an intra prediction mode, a Most Probable Mode (MPM)flag, and an MPM index. The intra prediction unit 252 predicts thesample values of the current block by using the reconstructed sampleslocated on the left and/or upper side of the current block as referencesamples. In this disclosure, reconstructed samples, reference samples,and samples of the current block may represent pixels. Also, samplevalues may represent pixel values.

According to an embodiment, the reference samples may be samplesincluded in a neighboring block of the current block. For example, thereference samples may be samples adjacent to a left boundary of thecurrent block and/or samples may be samples adjacent to an upperboundary. Also, the reference samples may be samples located on a linewithin a predetermined distance from the left boundary of the currentblock and/or samples located on a line within a predetermined distancefrom the upper boundary of the current block among the samples ofneighboring blocks of the current block. In this case, the neighboringblock of the current block may include the left (L) block, the upper (A)block, the below left (BL) block, the above right (AR) block, or theabove left (AL) block.

The inter prediction unit 254 generates a prediction block usingreference pictures and inter encoding information stored in the DPB 256.The inter coding information may include motion information set(reference picture index, motion vector information, etc.) of thecurrent block for the reference block. Inter prediction may include L0prediction, L1 prediction, and bi-prediction. L0 prediction meansprediction using one reference picture included in the L0 picture list,and L1 prediction means prediction using one reference picture includedin the L1 picture list. For this, one set of motion information (e.g.,motion vector and reference picture index) may be required. In thebi-prediction method, up to two reference regions may be used, and thetwo reference regions may exist in the same reference picture or mayexist in different pictures. That is, in the bi-prediction method, up totwo sets of motion information (e.g., a motion vector and a referencepicture index) may be used and two motion vectors may correspond to thesame reference picture index or different reference picture indexes. Inthis case, the reference pictures may be displayed (or outputted) bothbefore and after the current picture in time aspect. According to anembodiment, two reference regions used in the bi-prediction scheme maybe regions selected from picture list L0 and picture list L1,respectively.

The inter prediction unit 254 may obtain a reference block of thecurrent block using a motion vector and a reference picture index. Thereference block is in a reference picture corresponding to a referencepicture index. Also, a sample value of a block specified by a motionvector or an interpolated value thereof may be used as a predictor ofthe current block. For motion prediction with sub-pel unit pixelaccuracy, for example, an 8-tap interpolation filter for a luma signaland a 4-tap interpolation filter for a chroma signal may be used.However, the interpolation filter for motion prediction in sub-pel unitsis not limited thereto. In this way, the inter prediction unit 254performs motion compensation to predict the texture of the current unitfrom motion pictures reconstructed previously. In this case, the interprediction unit may use a motion information set.

According to an additional embodiment, the prediction unit 250 mayinclude an intra BC prediction unit (not shown). The intra BC predictionunit may reconstruct the current region by referring to a specificregion including reconstructed samples in the current picture. The intraBC prediction unit obtains intra BC encoding information for the currentregion from the entropy decoding unit 210. The intra BC prediction unitobtains a block vector value of the current region indicating thespecific region in the current picture. The intra BC prediction unit mayperform intra BC prediction by using the obtained block vector value.The intra BC encoding information may include block vector information.

The reconstructed video picture is generated by adding the predict valueoutputted from the intra prediction unit 252 or the inter predictionunit 254 and the residual value outputted from the inversetransformation unit 225. That is, the video signal decoding apparatus200 reconstructs the current block using the prediction block generatedby the prediction unit 250 and the residual obtained from the inversetransformation unit 225.

Meanwhile, the block diagram of FIG. 2 shows a decoding apparatus 200according to an embodiment of the present invention, and separatelydisplayed blocks logically distinguish and show the elements of thedecoding apparatus 200. Accordingly, the elements of the above-describeddecoding apparatus 200 may be mounted as one chip or as a plurality ofchips depending on the design of the device. According to an embodiment,the operation of each element of the above-described decoding apparatus200 may be performed by a processor (not shown).

FIG. 3 illustrates an embodiment in which a coding tree unit (CTU) issplit into coding units (CUs) in a picture. In the coding process of avideo signal, a picture may be split into a sequence of coding treeunits (CTUs). The coding tree unit is composed of an N×N block of lumasamples and two blocks of chroma samples corresponding thereto. Thecoding tree unit may be split into a plurality of coding units. Thecoding tree unit is not split and may be a leaf node. In this case, thecoding tree unit itself may be a coding unit. The coding unit refers toa basic unit for processing a picture in the process of processing thevideo signal described above, that is, intra/inter prediction,transformation, quantization, and/or entropy coding. The size and shapeof the coding unit in one picture may not be constant. The coding unitmay have a square or rectangular shape. The rectangular coding unit (orrectangular block) includes a vertical coding unit (or vertical block)and a horizontal coding unit (or horizontal block). In the presentspecification, the vertical block is a block whose height is greaterthan the width, and the horizontal block is a block whose width isgreater than the height. Further, in this specification, a non-squareblock may refer to a rectangular block, but the present invention is notlimited thereto.

Referring to FIG. 3, the coding tree unit is first split into a quadtree (QT) structure. That is, one node having a 2N×2N size in a quadtree structure may be split into four nodes having an N×N size. In thepresent specification, the quad tree may also be referred to as aquaternary tree. Quad tree split may be performed recursively, and notall nodes need to be split with the same depth.

Meanwhile, the leaf node of the above-described quad tree may be furthersplit into a multi-type tree (MTT) structure. According to an embodimentof the present invention, in a multi-type tree structure, one node maybe split into a binary or ternary tree structure of horizontal orvertical division. That is, in the multi-type tree structure, there arefour split structures such as vertical binary split, horizontal binarysplit, vertical ternary split, and horizontal ternary split. Accordingto an embodiment of the present invention, in each of the treestructures, the width and height of the nodes may all have powers of 2.For example, in a binary tree (BT) structure, a node of a 2N×2N size maybe split into two N×2N nodes by vertical binary split, and split intotwo 2N×N nodes by horizontal binary split. In addition, in a ternarytree (TT) structure, a node of a 2N×2N size is split into (N/2)×2N,N×2N, and (N/2)×2N nodes by vertical ternary split, and split into2N×(N/2), 2N×N, and 2N×(N/2) nodes by horizontal binary split. Thismulti-type tree split may be performed recursively.

The leaf node of the multi-type tree can be a coding unit. If splittingfor the coding unit is not indicated or the coding unit is not greaterthan the maximum transform length, the coding unit is used as a unit ofprediction and transform without further splitting. Meanwhile, in thequad tree and multi-type tree described above, at least one of thefollowing parameters can be defined in advance or can be transmittedthrough RBSP of a high level set such as PPS, SPS, VPS, and the like. 1)CTU size: root node size of quad tree, 2) Minimum QT size MinQtSize:Minimum QT leaf node size allowed, 3) Maximum BT size MaxBtSize:Mmaximum BT root node size allowed, 4) Maximum TT size MaxTtSize:Maximum TT root node size allowed, 5) Maximum MTT depth MaxMttDepth:Maximum allowable depth of MTT split from leaf node of QT, 6) Minimum BTsize MinBtSize: Minimum BT leaf node size allowed, 7) Minimum TT sizeMinTtSize: Minimum TT leaf node size allowed.

FIG. 4 shows an embodiment of a method for signaling the split of a quadtree and a multi-type tree. Preset flags may be used to signal the splitof the above-described quad tree and multi-type tree. Referring to FIG.4, at least one of a flag ‘qt_split_flag’ indicating whether to splitthe quad tree node, a flag ‘mtt_split_flag’ indicating whether to splitthe multi-type tree node, a flag ‘mtt_split_vertical_flag’ indicating asplit direction of a multi-type tree node, or a flag‘mtt_split_binary_flag’ indicating a split shape of a multi-type treenode may be used.

According to an embodiment of the present invention, the coding treeunit is a root node of a quad tree, and may be first split into a quadtree structure. In the quad tree structure, ‘qt_split_flag’ is signaledfor each node ‘QT_node’. If the value of ‘qt_split_flag’ is 1, the nodeis split into 4 square nodes, and if the value of ‘qt_split_flag’ is 0,the corresponding node becomes the leaf node ‘QT_leaf_node’ of the quadtree.

Each quad tree leaf node ‘QT_leaf_node’ may be further split into amulti-type tree structure. In the multi-type tree structure,‘mtt_split_flag’ is signaled for each node ‘MTT_node’. When the value of‘mtt_split_flag’ is 1, the corresponding node is split into a pluralityof rectangular nodes, and when the value of ‘mtt_split_flag’ is 0, thecorresponding node is a leaf node ‘MTT_leaf_node’ of the multi-typetree. When the multi-type tree node ‘MTT_node’ is split into a pluralityof rectangular nodes (i.e., when the value of ‘mtt_split_flag’ is 1),‘mtt_split_vertical_flag’ and ‘mtt_split_binary_flag’ for the node‘MTT_node’ may be additionally signaled. When the value of‘mtt_split_vertical_flag’ is 1, vertical split of node ‘MTT_node’ isindicated, and when the value of ‘mtt_split_vertical_flag’ is 0,horizontal split of node ‘MTT_node’ is indicated. In addition, when thevalue of ‘mtt_split_binary_flag’ is 1, the node ‘MTT_node’ is split into2 rectangular nodes, and when the value of ‘mtt_split_binary_flag’ is 0,the node ‘MTT_node’ is split into 3 rectangular nodes.

Picture prediction (motion compensation) for coding is performed on acoding unit that is no longer divided (i.e., a leaf node of a codingunit tree). Hereinafter, the basic unit for performing the predictionwill be referred to as a “prediction unit” or a “prediction block”.

Hereinafter, the term “unit” used herein may replace the predictionunit, which is a basic unit for performing prediction. However, thepresent disclosure is not limited thereto, and “unit” may be understoodas a concept broadly encompassing the coding unit.

FIGS. 5 and 6 more specifically illustrate an intra prediction methodaccording to an embodiment of the present invention. As described above,the intra prediction unit predicts the sample values of the currentblock by using the reconstructed samples located on the left and/orupper side of the current block as reference samples.

First, FIG. 5 shows an embodiment of reference samples used forprediction of a current block in an intra prediction mode. According toan embodiment, the reference samples may be samples adjacent to the leftboundary of the current block and/or samples adjacent to the upperboundary. As shown in FIG. 5, when the size of the current block is W×Hand samples of a single reference line adjacent to the current block areused for intra prediction, reference samples may be configured using amaximum of 2W+2H+1 neighboring samples located on the left and/or upperside of the current block.

When at least some samples to be used as reference samples have not yetbeen reconstructed, the intra prediction unit may obtain referencesamples by performing a reference sample padding procedure. The intraprediction unit may perform a reference sample filtering procedure toreduce an error in intra prediction. That is, filtering may be performedon neighboring samples and/or reference samples obtained by thereference sample padding procedure, so as to obtain the filteredreference samples. The intra prediction unit predicts samples of thecurrent block by using the reference samples obtained as in the above.The intra prediction unit predicts samples of the current block by usingunfiltered reference samples or filtered reference samples. In thepresent disclosure, neighboring samples may include samples on at leastone reference line. For example, the neighboring samples may includeadjacent samples on a line adjacent to the boundary of the currentblock.

Next, FIG. 6 shows an embodiment of prediction modes used for intraprediction. For intra prediction, intra prediction mode informationindicating an intra prediction direction may be signaled. The intraprediction mode information indicates one of a plurality of intraprediction modes included in the intra prediction mode set. When thecurrent block is an intra prediction block, the decoder receives intraprediction mode information of the current block from the bitstream. Theintra prediction unit of the decoder performs intra prediction on thecurrent block based on the extracted intra prediction mode information.

According to an embodiment of the present invention, the intraprediction mode set may include all intra prediction modes used in intraprediction (e.g., a total of 67 intra prediction modes). Morespecifically, the intra prediction mode set may include a planar mode, aDC mode, and a plurality (e.g., 65) of angle modes (i.e., directionalmodes). Each intra prediction mode may be indicated through a presetindex (i.e., intra prediction mode index). For example, as shown in FIG.6, the intra prediction mode index 0 indicates a planar mode, and theintra prediction mode index 1 indicates a DC mode. Also, the intraprediction mode indexes 2 to 66 may indicate different angle modes,respectively. The angle modes respectively indicate angles which aredifferent from each other within a preset angle range. For example, theangle mode may indicate an angle within an angle range (i.e., a firstangular range) between 45 degrees and −135 degrees clockwise. The anglemode may be defined based on the 12 o'clock direction. In this case, theintra prediction mode index 2 indicates a horizontal diagonal (HDIA)mode, the intra prediction mode index 18 indicates a horizontal(Horizontal, HOR) mode, the intra prediction mode index 34 indicates adiagonal (DIA) mode, the intra prediction mode index 50 indicates avertical (VER) mode, and the intra prediction mode index 66 indicates avertical diagonal (VDIA) mode.

Meanwhile, the preset angle range may be set differently depending on ashape of the current block. For example, if the current block is arectangular block, a wide angle mode indicating an angle exceeding 45degrees or less than −135 degrees in a clockwise direction may beadditionally used. When the current block is a horizontal block, anangle mode may indicate an angle within an angle range (i.e., a secondangle range) between (45+offset1) degrees and (−135+offset1) degrees ina clockwise direction. In this case, angle modes 67 to 76 outside thefirst angle range may be additionally used. In addition, if the currentblock is a vertical block, the angle mode may indicate an angle withinan angle range (i.e., a third angle range) between (45−offset2) degreesand (−135−offset2) degrees in a clockwise direction. In this case, anglemodes −10 to −1 outside the first angle range may be additionally used.According to an embodiment of the present invention, values of offset1and offset2 may be determined differently depending on a ratio betweenthe width and height of the rectangular block. In addition, offset1 andoffset2 may be positive numbers.

According to a further embodiment of the present invention, a pluralityof angle modes configuring the intra prediction mode set may include abasic angle mode and an extended angle mode. In this case, the extendedangle mode may be determined on the basis of the basic angle mode.

According to an embodiment, the basic angle mode is a mode correspondingto an angle used in intra prediction of the existing high efficiencyvideo coding (HEVC) standard, and the extended angle mode may be a modecorresponding to an angle newly added in intra prediction of the nextgeneration video codec standard. More specifically, the basic angle modemay be an angle mode corresponding to any one of the intra predictionmodes {2, 4, 6, 66}, and the extended angle mode may be an angle modecorresponding to any one of the intra prediction modes {3, 5, 7, . . . ,65}. That is, the extended angle mode may be an angle mode between basicangle modes within the first angle range. Accordingly, the angleindicated by the extended angle mode may be determined on the basis ofthe angle indicated by the basic angle mode.

According to another embodiment, the basic angle mode may be a modecorresponding to an angle within a preset first angle range, and theextended angle mode may be a wide angle mode outside the first anglerange. That is, the basic angle mode may be an angle mode correspondingto any one of the intra prediction modes {2, 3, 4, . . . , 66}, and theextended angle mode may be an angle mode corresponding to any one of theintra prediction modes {−10, −9, . . . , −1} and {67, 68, . . . , 76}.The angle indicated by the extended angle mode may be determined as anangle on a side opposite to the angle indicated by the correspondingbasic angle mode. Accordingly, the angle indicated by the extended anglemode may be determined on the basis of the angle indicated by the basicangle mode. Meanwhile, the number of extended angle modes is not limitedthereto, and additional extended angles may be defined according to thesize and/or shape of the current block. For example, the extended anglemode may be defined as an angle mode corresponding to any one of theintra prediction modes {−14, −13, . . . , −1} and {67, 68, . . . , 80}.Meanwhile, the total number of intra prediction modes included in theintra prediction mode set may vary depending on the configuration of thebasic angle mode and extended angle mode described above.

In the embodiments described above, the spacing between the extendedangle modes may be set on the basis of the spacing between thecorresponding basic angle modes. For example, the spacing between theextended angle modes {3, 5, 7, . . . , 65} may be determined on thebasis of the spacing between the corresponding basic angle modes {2, 4,6, . . . , 66}. In addition, the spacing between the extended anglemodes {−10, −9, . . . , −1} may be determined on the basis of thespacing between corresponding basic angle modes {56, 57, . . . , 65} onthe opposite side, and the spacing between the extended angle modes {67,68, . . . , 76} may be determined on the basis of the spacing betweenthe corresponding basic angle modes {3, 4, . . . , 12} on the oppositeside. The angular spacing between the extended angle modes may be set tobe the same as the angular spacing between the corresponding basic anglemodes. In addition, the number of extended angle modes in the intraprediction mode set may be set to be less than or equal to the number ofbasic angle modes.

According to an embodiment of the present invention, the extended anglemode may be signaled on the basis of the basic angle mode. For example,the wide angle mode (i.e., the extended angle mode) may replace at leastone angle mode (i.e., the basic angle mode) within the first anglerange. The basic angle mode to be replaced may be a corresponding anglemode on a side opposite to the wide angle mode. That is, the basic anglemode to be replaced is an angle mode that corresponds to an angle in anopposite direction to the angle indicated by the wide angle mode or thatcorresponds to an angle that differs by a preset offset index from theangle in the opposite direction. According to an embodiment of thepresent invention, the preset offset index is 1. The intra predictionmode index corresponding to the basic angle mode to be replaced may beremapped to the wide angle mode to signal the corresponding wide anglemode. For example, the wide angle modes {−10, −9, . . . , −1} may besignaled by the intra prediction mode indices {57, 58, . . . , 66},respectively, and the wide angle modes {67, 68, . . . , 76} may besignaled by the intra prediction mode indices {2, 3, . . . , 11},respectively. In this way, the intra prediction mode index signals theextended angle mode, and thus the same set of intra prediction modeindices may be used for signaling of the intra prediction mode even ifthe configuration of the angle modes used for intra prediction of eachblock are different from each other. Accordingly, signaling overhead dueto a change in the intra prediction mode configuration may be minimized.

Meanwhile, whether or not to use the extended angle mode may bedetermined on the basis of at least one of the shape and size of thecurrent block. According to an embodiment, if the size of the currentblock is greater than a preset size, the extended angle mode may be usedfor intra prediction of the current block, otherwise, only the basicangle mode may be used for intra prediction of the current block.According to another embodiment, if the current block is a block otherthan a square, the extended angle mode may be used for intra predictionof the current block, and if the current block is a square block, onlythe basic angle mode may be used for intra prediction of the currentblock.

In the following, an inter prediction method according to an embodimentof the present invention will be described with reference to FIG. 7. Inthe present disclosure, the inter prediction method may include ageneral inter prediction method optimized for translation motion and anaffine model-based inter prediction method. In addition, the motionvector may include at least one of a general motion vector for motioncompensation according to the general inter prediction method and acontrol point motion vector for affine motion compensation.

FIG. 7 shows an inter prediction method according to an embodiment ofthe present invention. As described above, the decoder may predict thecurrent block by referring to reconstructed samples of another decodedpicture. Referring to FIG. 7, the decoder obtains a reference block 702in a reference picture 720 on the basis of the motion information set ofthe current block 701. In this case, the motion information set mayinclude a reference picture index and a motion vector. The referencepicture index indicates the reference picture 720 including a referenceblock for inter prediction of the current block in the reference picturelist. According to an embodiment, the reference picture list may includeat least one of the L0 picture list and L1 picture list described above.The motion vector represents an offset between a coordinate value of thecurrent block 701 in the current picture 710 and a coordinate value ofthe reference block 702 in the reference picture 720. The decoderobtains a predictor of the current block 701 on the basis of samplevalues of the reference block 702, and reconstructs the current block701 using the predictor.

Specifically, the encoder may obtain the reference block described aboveby searching for a block similar to the current block from pictures ofwhich the reconstruction order is earlier. For example, the encoder maysearch for a reference block for which the sum of the difference betweenthe current block and the sample value is minimum within a preset searchregion. In this case, in order to measure the similarity between thecurrent block and the samples of the reference block, at least one ofsum of absolute difference (SAD) or sum of Hadamard transformeddifference (SATD) may be used. Here, SAD may be a value obtained bysumming the absolute values of respective differences between samplevalues included in the two blocks. In addition, SATD may be a valueobtained by adding all absolute values of Hadamard transformcoefficients obtained by performing Hadamard transform on the differencebetween sample values included in two blocks.

Meanwhile, the current block may be predicted using one or morereference regions. As described above, the current block may beinter-predicted through a bi-prediction method using two or morereference regions. According to an embodiment, the decoder may obtaintwo reference blocks on the basis of two motion information sets of thecurrent block. In addition, the decoder may obtain a first predictor anda second predictor of the current block on the basis of the respectivesample values of the obtained two reference blocks. In addition, thedecoder may reconstruct the current block using the first predictor andthe second predictor. For example, the decoder may reconstruct thecurrent block on the basis of the mean for each sample of the firstpredictor and the second predictor.

As described above, one or more motion information sets may be signaledfor motion compensation of the current block. In this case, similaritybetween motion information sets for motion compensation of each of theplurality of blocks may be used. For example, the motion information setused for prediction of the current block may be induced from the motioninformation set used for prediction of any one of the previouslyreconstructed other samples. Through this, the encoder and decoder mayreduce signaling overhead.

For example, a plurality of candidate blocks may exist that are likelyto have been predicted on the basis of the same or similar motioninformation set as the motion information set of the current block. Thedecoder may generate a merge candidate list on the basis of thecorresponding plurality of candidate blocks. Here, the merge candidatelist may include candidates corresponding to samples that are likely tohave been predicted on the basis of a motion information set related tothe motion information set of the current block, among samplesreconstructed earlier than the current block. The encoder and decodermay configure a merge candidate list of the current block according to apredefined rule. In this case, the merge candidate lists respectivelyconfigured by the encoder and the decoder may be the same. For example,the encoder and decoder may construct a merge candidate list of thecurrent block on the basis of a position of the current block in thecurrent picture. A method of configuring the merge candidate list of thecurrent block by the encoder and the decoder will be described laterwith reference to FIG. 9. In the present disclosure, a position of aspecific block indicates a relative position of a top-left sample of thespecific block within a picture including the specific block.

Meanwhile, in order to improve coding efficiency, a method of quantizinga transform coefficient value obtained by transforming a residual signaland coding the quantized transform coefficient may be used instead ofcoding the above-described residual as it is. As described above, thetransform unit may obtain a transform coefficient value by transforminga residual signal. In this case, the residual signal of a specific blockmay be distributed over an entire area of the current block.Accordingly, it is possible to improve coding efficiency byconcentrating energy in the low frequency region through frequencydomain conversion of a residual signal. Hereinafter, a method oftransforming or inversely transforming a residual signal will bedescribed in detail.

FIG. 8 is a diagram specifically illustrating a method for transforminga residual signal by an encoder. As described above, a residual signalin a spatial domain may be transformed to a frequency domain. An encodermay obtain a transform coefficient by transforming the obtained residualsignal. First, the encoder may obtain at least one residual blockincluding a residual signal for a current block. The residual block maybe either the current block or one of blocks divided from the currentblock. In the present disclosure, the residual block may be referred toas a residual array or a residual matrix which include residual samplesof the current block. In the present disclosure, the residual block mayrepresent a transform unit or a block having the same size as that ofthe transform block.

Next, the encoder may transform the residual block by using a transformkernel. The transform kernel used for transformation of the residualblock may be a transform kernel having separable characteristics ofvertical transform and horizontal transform. In this case, the transformfor the residual block may be performed separately into verticaltransform and horizontal transform. For example, the encoder may performvertical transformation by applying a transform kernel in the verticaldirection of the residual block. The encoder may perform horizontaltransform by applying the transform kernel in the horizontal directionof the residual block. In the present disclosure, the transform kernelmay be used as a term to refer to a parameter set used for transform ofthe residual signal, such as transform matrix, transform array,transform function, and transform. According to an embodiment, thetransform kernel may be any one of a plurality of available kernels. Atransform kernel based on different transform types may be used for eachof the vertical transform and the horizontal transform.

The encoder may transfer the transform block transformed from theresidual block to a quantization unit and quantize the transform block.The transform block may include a plurality of transform coefficients.Specifically, the transform block may include the plurality of transformcoefficients arranged in two dimensions. As in the case of the residualblock, the size of the transform block may be the same as the size ofeither the current block or the block divided from the current block.The transform coefficients transferred to the quantization unit may beexpressed as quantized values.

The encoder may perform additional transform before the transformcoefficients are quantized. As illustrated in FIG. 8, theabove-described transform method may be referred to as a primarytransform, and an additional transform may be referred to as a secondarytransform. The secondary transform may be selective for each residualblock. According to an embodiment, the encoder may improve codingefficiency by performing secondary transform for a region where it isdifficult to concentrate energy in a low-frequency region only byprimary transform. For example, secondary transform may be added to ablock in which residual values appear larger in a direction other thanthe horizontal or vertical direction of the residual block. The residualvalues of an intra-predicted block may have a higher probability oftransformation in a direction other than the horizontal or verticaldirection compared to the residual values of an inter-predicted block.Accordingly, the encoder may additionally perform secondary transform onthe residual signal of the intra-predicted block. The encoder may omitsecondary transform for the residual signal of the inter-predictedblock.

As another example, whether or not to perform the secondary transformmay be determined depending on the size of the current block or theresidual block. In addition, transform kernels having different sizesmay be used depending on the size of the current block or the residualblock. For example, 8×8 secondary transform may be applied to a blockwhose length of a shorter side of the width and height is greater thanor equal to a first preset length. In addition, 4×4 secondary transformmay be applied to a block whose length of the shorter side of the widthand height is greater than or equal to a second preset length and lessthan the first preset length. In this case, the first preset length maybe a value greater than the second preset length, but the presentdisclosure is not limited thereto. In addition, unlike primarytransform, secondary transform may not be performed separately intovertical transform and horizontal transform. This secondary transformmay be referred to as a low frequency non-separable transform (LFNST).

In the case of a video signal in a specific region, energy in a highfrequency band may not be reduced even if frequency transformation isperformed due to a sudden change in brightness. Accordingly, compressionperformance due to quantization may be deteriorated. When transform isperformed on a region in which a residual value rarely exists, anencoding time and a decoding time may be unnecessarily increased.Accordingly, transform on the residual signal of the specific region maybe omitted. Whether to perform transform on the residual signal of thespecific region may be determined by a syntax element related totransform of the specific region. For example, the syntax element mayinclude transform skip information. The transform skip information maybe a transform skip flag. If the transform skip information on theresidual block indicates a transform skip, transform on the residualblock is not performed. In this case, the encoder may immediatelyquantize the residual signal on which transform of a correspondingregion has not been performed. The operations of the encoder describedwith reference to FIG. 8 may be performed via the transform unit of FIG.1.

The above-described transform-related syntax elements may be informationparsed from a video signal bitstream. The decoder may entropy-decode thevideo signal bitstream so as to obtain transform-related syntaxelements. The encoder may entropy-code the transform-related syntaxelements so as to generate a video signal bitstream.

FIG. 9 is a diagram specifically illustrating a method for obtaining aresidual signal by inverse transforming a transformation coefficient byan encoder and a decoder. For the convenience of description, it will bedescribed that an inverse transform operation is performed via aninverse transform unit of each of an encoder and a decoder. The inversetransform unit may obtain a residual signal by inverse transforming aninverse quantized transform coefficient. First, the inverse transformunit may detect whether inverse transform for a specific region isperformed, from a transform-related syntax element of the region.According to an embodiment, when a transform-related syntax element fora specific transform block indicates a transform skip, transform on thetransform block may be omitted. In this case, both the primary inversetransform and the secondary inverse transform described above regardingthe transform block may be omitted. The inverse quantized transformcoefficient may be used as a residual signal. For example, the decodermay reconstruct a current block by using the inverse quantized transformcoefficient as a residual signal.

According to another embodiment, the transform-related syntax elementfor the specific transform block may not indicate a transform skip. Inthis case, the inverse transform unit may determine whether to performsecondary inverse transform for secondary transform. For example, whenthe transform block is a transform block of an intra-predicted block,secondary inverse transform may be performed on the transform block. Asecondary transform kernel used for the transform block may bedetermined based on an intra prediction mode corresponding to thetransform block. As another example, whether to perform secondaryinverse transform may be determined based on the size of the transformblock. Secondary inverse transform may be performed after inversequantization and before primary inverse transform.

The inverse transform unit may perform primary inverse transform on theinverse quantized transform coefficient or a secondary inversetransformed transform coefficient. In the case of primary inversetransform, vertical transform and horizontal transform may be performedseparately as in the case of primary transform. For example, the inversetransform unit may obtain a residual block by performing verticalinverse transform and horizontal inverse transform on the transformblock. The inverse transform unit may inverse transform the transformblock on the basis of the transform kernel used for transforming thetransform block. For example, the encoder may explicitly or implicitlysignal information indicating the transform kernel applied to thecurrent transform block from among a plurality of available transformkernels. The decoder may select a transform kernel to be used forinverse transform of the transform block from among the plurality ofavailable transform kernels by using information indicating the signaledtransform kernel. The inverse transform unit may reconstruct the currentblock by using the residual signal obtained via inverse transform on thetransform coefficient.

Meanwhile, a distribution of the residual signals of a picture may bedifferent for each region. For example, a distribution of values for theresidual signal in a specific region may vary depending on a predictionmethod. If transform is performed on a plurality of different transformregions using the same transform kernel, coding efficiency may vary foreach transform region depending on distributions and characteristics ofvalues in the transform regions. Accordingly, if a transform kernel usedfor transform of specific transform block is adaptively selected fromamong a plurality of transform kernels that may be used, codingefficiency may be further improved. That is, the encoder and decoder mayadditionally be configured to use a transform kernel other than thebasic transform kernel in transforming a video signal. A method ofadaptively selecting the transform kernel may be referred to as adaptivemultiple core transform (AMT) or multiple transform selection (MTS). Themethod of adaptively selecting the transform kernel will be describedlater with reference to related drawings. In the present disclosure, forconvenience of description, transform and inverse transform arecollectively referred to as transform. In addition, the transform kerneland the inverse transform kernel are collectively referred to as atransform kernel.

In the following, a plurality of transform kernels that may be used fortransform of a video signal will be described with reference to FIG. 10.According to an embodiment of the present invention, the transformkernel may be a kernel derived on the basis of a specific basisfunction. The plurality of different transform kernels may berespectively obtained based on different basis functions. The pluralityof transform kernels may be obtained on the basis of basis functionsrespectively corresponding to different transform types.

FIG. 10 is a diagram representing a basis function corresponding to eachtransform type. According to an embodiment, a transform kernel may beused for transform of a video residual signal may include at least oneof a discrete cosine transform type-II (DCT-II)-based transform kernel,a discrete cosine transform type-V (DCT-V)-based transform kernel, adiscrete cosine transform type-VIII (DCT-VIII)-based transform kernel, adiscrete sine transform type-I (DST-I)-based transform kernel, and adiscrete sine transform type-II (DST-VII)-based transform kernel.

Referring to FIG. 10, the basis functions respectively corresponding tothe transform types DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VIIdescribed above may be expressed as a function of cosine or sine. Forexample, the basis functions respectively corresponding to DCT-II,DCT-V, and DCT-VIII may be a cosine type function, and the basisfunctions respectively corresponding to DST-I and DST-VII may be a sinetype function. In addition, a basis function corresponding to a specifictransform type may be represented in the form of a basis function foreach frequency bin. For example, a basis function Ti(j) may be a basisfunction corresponding to an i-th frequency bin. That is, as the valuerepresented by i is smaller, the basis function corresponding to a lowerfrequency is represented. In addition, as the value indicated by i islarger, the basis function corresponding to a higher frequency isrepresented. In FIG. 10, j may represent an integer between 0 and N−1.

The basis function Ti(j) may be expressed as a two-dimensional matrixrepresenting a j-th element of an i-th row. In this case, the transformusing transform kernels on the basis of the transform type of FIG. 9 hasseparable characteristics in performing transform. That is, thetransform for each of a horizontal direction and a vertical directionmay be performed separately for the residual signal. For example, thetransform using a transform matrix T for a residual block X may berepresented as a matrix operation TXT′. In this case, T′ means atranspose matrix of the transform matrix T.

Values of the transform matrix calculated by the basis function shown inFIG. 10 may be in a decimal form rather than an integer form. It may bedifficult to implement values in the form of decimal values in hardwarein a video encoding device and decoding device. Accordingly, a transformkernel approximated by an integer from an original transform kernelincluding values in the form of decimals may be used for encoding anddecoding of a video signal. An approximated transform kernel includingvalues in the form of integer values may be generated through scalingand rounding of the original transform kernel. The integer valueincluded in the approximated transform kernel may be a value within arange that may be expressed by a preset number of bits. The presetnumber of bits may be an 8-bit or 10-bit. Depending on theapproximation, an orthonormal property of DCT and DST may not bemaintained. However, the encoding efficiency loss due to theapproximation is not large, it may be advantageous in terms of hardwareimplementation to approximate the transform kernel in an integer form.

FIG. 11 is a diagram representing a magnitude of a signal for each indexof each of transform types DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII.FIG. 11 shows a form of a basis function corresponding to the lowestfrequency bin among basis functions for each frequency bin. FIG. 11represents a basis function corresponding to a 0-th frequency bin amongbasis functions for each frequency bin. In FIG. 11, the horizontal axisrepresents an index j (j=0, 1, . . . , N−1) in the basis function, andthe vertical axis represents a magnitude value of a signal. N representsthe number of samples in a specific region targeted for transform.

As shown in FIG. 11, in DST-VII, the magnitude of the signal tends toincrease as the index j increases. Accordingly, DST-VII, like a residualblock of an intra-predicted block, may be efficient for transform of aresidual block in which the magnitude of the residual signal increasesas the distance in the horizontal and vertical directions increases,with the top-left of the residual block as a reference.

On the other hand, in DCT-VIII, the magnitude of the signal tends todecrease as the index j increases. That is, DCT-VIII satisfies dualitycharacteristics along with DST-VII. Accordingly, DCT-VIII may beefficient for transform of a residual block in which the magnitude ofthe residual signal decreases as the distance in the horizontal andvertical directions increases with the top-left of the residual block asa reference.

In DST-I, as the index j in the basis function increases, the magnitudeof the signal increases, and then the magnitude of the signal decreasesstarting from a specific index. Therefore, it may be efficient fortransform of a residual block having a large size of the residual signalat the center portion of the residual block.

The 0-th basis function of DCT-II represents DC. Accordingly, it may beefficient for a residual block having a uniform signal magnitudedistribution in the residual block.

DCT-V is similar to DCT-II, but the magnitude of the signal for a casewhere the index j is 0 is less than the magnitude of the signal for acase where the index j is not 0. That is, if the index j is 1, it has asignal model in which the straight line is bent.

Meanwhile, as described above, when a transform kernel having theseparable characteristics is used, transform may be performed for eachof the horizontal direction and the vertical direction of the residualblock. Specifically, the transform for the residual block may beperformed through two times of 2D matrix multiplication operation. Thematrix multiplication operation may be accompanied by an amount ofcalculation greater than or equal to a preset level. Accordingly, whenthe residual block is transformed using a DCT-II-based transform kernel,the amount of calculation may be reduced by using a combinationstructure of a butterfly structure or a half butterfly structure and ahalf matrix multiplier. However, DST-VII and DCT-VIII have relativelyhigh implementation complexity, and thus implementation may be difficultin terms of the amount of calculation. In addition, the elements of thetransform kernel (matrix elements of the transform kernel) should bestored in a memory for operation, a memory capacity for storing thekernel matrix should also be considered in implementation. Accordingly,a transform type having similar characteristics to each of DST-VII andDCT-VIII and relatively low implementation complexity may be used.

According to an embodiment, discrete sine transform type IV (DST-IV) anddiscrete cosine transform type-IV (DCT-IV) may replace DST-VII andDCT-VIII, respectively.

FIG. 12 is a diagram representing the magnitude of a signal for eachindex for each of the transform types DST-IV, DCT-IV, DST-VII, andDCT-VIII. FIG. 12(a) represents a basis function corresponding to DST-IVand a basis function corresponding to DCT-IV. DST-IV and DCT-IV for thenumber N of samples may be derived from DCT-II for the number 2N ofsamples. That is, a DCT-II partial butterfly structure for the number 2Nof samples includes DCT-IV for the number N of samples. In addition,DST-IV for the number N of samples may be implemented by arranging asign inversion operation and the corresponding basis function in reverseorder from DCT-IV for the number N of samples.

As shown in FIG. 12(b), DST-IV represents a signal model similar toDST-VII. Accordingly, DST-IV, like a residual block of anintra-predicted block, may be efficient for transform of a residualblock in which the magnitude of the residual signal increases as thedistance in the horizontal and vertical directions increases, with thetop-left of the residual block as a reference. In addition, DCT-IVrepresents a signal model similar to DCT-VIII. Accordingly, DCT-IV maybe efficient for transform of a residual block in which the magnitude ofthe residual signal is large at a specific boundary, and the magnitudeof the residual signal decreases as it goes from the correspondingboundary to another boundary.

If only a transform kernel based on any one of the transform typesdescribed above is used for transform of a video signal, it is difficultto perform adaptive transform according to a pattern of the residualsignal, which varies depending on the characteristics of the originalsignal itself and the prediction mode. Accordingly, the encoder anddecoder according to an embodiment of the present invention may improvecoding efficiency by using a transform kernel selected for each regionfrom among a plurality of usable transform kernels in transforming aresidual signal.

In the existing high efficiency video coding (HEVC) standard, among theplurality of transform types described above, the residual signal wastransformed using the DCT-II-based transform kernel and the residualsignal was transformed using the DST-VII-based transform kernellimitedly for an intra-predicted block having a 4×4 size. As describedabove, DCT-II may be suitable for transform of the residual signal ofthe inter-predicted block, but may not be suitable for transform of theresidual signal of the intra-predicted block. That is, depending on howthe current block is predicted, the pattern of the residual may varywithin the residual block.

Accordingly, at least one selected from among a plurality of usabletransform kernels may be used for transform of a specific transformblock. According to an embodiment, the encoder and decoder may select atransform kernel for at least one transform region of the current blockon the basis of how the current block is predicted. Such a method may bereferred to as a multiple core transform (AMT) described above. If atransform kernel determined according to a prediction method for aspecific region is used for transform of a corresponding region, codingefficiency may be improved compared to a case in which transform isperformed using the DCT-II-based transform kernel for all regions.

FIG. 13 is a diagram representing a method of configuring a transformcandidate set according to a prediction mode of a current block.According to an embodiment of the present invention, the encoder and thedecoder may select a transform kernel used for at least one transformregion of the current block on the basis of the transform candidate setcorresponding to the current block. The transform candidate set mayinclude a plurality of candidate transform kernels. In addition, thetransform candidate set may include different candidate transformkernels according to the prediction mode of the current block.

According to an embodiment, the current block may be a block predictedon the basis of any one of a plurality of intra prediction modes. Inthis case, the pattern of the residual signal of the current block mayvary depending on the intra prediction mode used for prediction of thecurrent block. As described above, the intra prediction mode informationmay indicate an intra prediction direction. Accordingly, the pattern ofthe residual signal of the current block may vary depending on theprediction direction indicated by the intra prediction mode informationof the current block. The encoder and the decoder may improve codingefficiency by using a plurality of transform candidate sets that aredifferent depending on the prediction direction.

Accordingly, the encoder and decoder may determine a transform kernel tobe used for the transform region of the current block from the transformcandidate set corresponding to the intra prediction mode of the currentblock. For example, the transform kernel to be used for the transformregion may be signaled through a candidate index indicating any one ofthe plurality of candidate transform kernels included in the transformcandidate set. If the transform candidate set corresponding to the intraprediction mode of the current block is configured with two candidatetransform kernels, the candidate index indicating the transform kernelused for the transform region may be expressed with 1-bit.

In addition, different transform candidate sets may be used for thevertical and horizontal directions of the transform region,respectively. Due to the characteristics of the intra prediction methodpredicted on the basis of the directionality, a vertical directionpattern and a horizontal direction pattern of the residual signal of apredicted region may be different from each other. Accordingly, avertical transform kernel and a horizontal transform kernel for a blockpredicted using a specific intra prediction mode may be obtained fromindividual transform candidate sets. In the present disclosure, atransform kernel used for vertical transform of the specific transformregion may be referred to as a vertical transform kernel. In addition, atransform kernel used for horizontal transform of the specific transformregion may be referred to as a horizontal transform kernel. FIG. 13(a)represents sets of transform candidates, each set corresponding to eachof 67 intra prediction mode indexes. In FIG. 13(a), V (vertical)represents a transform candidate set for the vertical transform kernel.In addition, H (horizontal) represents a transform candidate set for thehorizontal transform kernel.

FIG. 13(b) represents a candidate transform kernel defined according toa transform candidate set. According to an embodiment, a first transformcandidate set (e.g., Transform Set 0 in FIG. 13(b)) may include aDST-VII-based candidate transform kernel and a DCT-VIII-based candidatetransform kernel. In addition, a second transform candidate set (e.g.,Transform Set 1 of FIG. 13(b)) may include a DST-VII-based candidatetransform kernel and a DST-I-based candidate transform kernel. A thirdtransform candidate set (e.g., Transform Set 2 in FIG. 13(b)) mayinclude a DST-VII-based candidate transform kernel and a DCT-V-basedcandidate transform kernel. FIG. 13 illustrates that the transformcandidate set includes two candidate transform kernels, but the presentdisclosure is not limited thereto.

According to another embodiment, the current block may be aninter-predicted block. In this case, the transform kernel used for thetransform region of the current block may be obtained from a presettransform candidate set. For example, the preset transform candidate setmay be any one of the plurality of transform candidate sets describedabove. FIG. 13(c) represents a transform candidate set corresponding tothe current block if the current block is an inter prediction block. Forexample, the preset transform candidate set corresponding to the interprediction block may include the DCT-VIII-based candidate transformkernel and the DST-VII-based candidate transform kernel. That is, thetransform kernel used for the transform region of the inter predictionblock may be any one of the DCT-VIII-based candidate transform kerneland the DST-VII-based candidate transform kernel.

According to an additional embodiment, the embodiments described abovemay be limitedly applied only to a transform block of a luma component.In this case, a preset basic transform kernel may be used for atransform block of a chroma component. For example, the basic transformkernel may be the DCT-II-based transform kernel.

According to an additional embodiment, the encoder may not signal thecandidate index depending on the number of non-zero transformcoefficients in the transform region. In this case, the decoder maydetermine the transform kernel by comparing the number of non-zerotransform coefficients in the transform region with a preset number. Forexample, if the number of non-zero transform coefficients is one or two,the candidate index may not be signaled. In this case, the inversetransformation unit may perform transform using a transform kernelpreset for the corresponding region. The preset transform kernel may bethe DST-VII-based transform kernel.

According to an additional embodiment, MTS may be applied only to theluma component, and DCT-II transform may be used for the chromacomponent, like HEVC. As an embodiment, on/off may be indicated with a1-bit flag so that MTS may be controlled in units of coding units, andif this flag indicates off, DCT-II, which is a basic kernel, may beused, like the chroma component. On the other hand, if this flagindicates on, a transform candidate index used in the preset transformset may be signaled according to the prediction mode and the inversetransform may be performed by applying a transform kernel correspondingto the index in the decoder. Different transforms may be applied to thehorizontal and vertical directions, a transform index to be used may beindicated with a total of 2-bit of which each 1-bit for each of thevertical and horizontal directions. Alternatively, the transform indexmay be indicated using a truncated unary binarization method. Forexample, if two DST-VII and DCT-VIII-based kernels may be used, fourcombinations of transform kernels applicable to the current block may bedefined, and may be signaled as follows.

{Horizontal: DST-VII, Vertical: DST-VII}, 0

{Horizontal: DCT-VIII, Vertical: DST-VII}, 10

{Horizontal: DST-VII, Vertical: DCT-VIII}, 110

{Horizontal: DCT-VIII, Vertical: DCT-VIII}, 111

According to an embodiment of the present invention, there is a highprobability that the combination {horizontal: DST-VII, vertical:DST-VII} is statistically selected in the MTS, if the transform kernelis signaled using the truncated unary binarization method describedabove rather than signaling the transform kernel using a fixed length of2-bit, an improvement in encoding efficiency may be expected.

FIG. 14 represents a horizontal transform kernel and a verticaltransform kernel determined according to a set index. According to anembodiment of the present invention, information indicating a transformkernel for a residual signal of a specific region among a plurality ofusable transform kernels may be signaled. In this case, informationindicating any one of the plurality of usable transform kernels may besignaled regardless of the prediction mode corresponding to thecorresponding region. For example, the encoder may signal transformkernel information indicating the transform kernel for the residualsignal in the specific region. The decoder may obtain a transform kernelfor the corresponding region by using the signaled transform kernelinformation. In this case, the transform kernel information may includeat least one of information representing the horizontal transform kernelof the corresponding region or information representing the verticaltransform kernel of the corresponding region. This method may bereferred to as the multiple transform selection (MTS) described above.

The transform kernel may be indicated by any one of kernel indexesrespectively representing the plurality of usable transform kernels.According to an embodiment, the transform kernel corresponding to thekernel index ‘0’ may be the DCT-II-based transform kernel, the transformkernel corresponding to the kernel index ‘1’ may be the DST-VII-basedtransform kernel, and the transform kernel corresponding to the kernelindex ‘2’ may be the DCT-VIII-based transform kernel.

According to an embodiment, the transform kernel information may be aset index representing the transform kernel set. The transform kernelset may represent a combination of a transform kernel to be used as thevertical transform kernel and a transform kernel to be used as thehorizontal transform kernel. In addition, the set index may indicate acombination of a kernel index indicating the horizontal transform kerneland a kernel index indicating the vertical transform kernel. Referringto FIG. 14(a), if the transform kernel information of a specific regionindicates a set index indicating a first transform kernel set (e.g., iftu_mts_idx[x0][y0] in FIG. 14(a) is 0), the vertical transform kernel ofthe corresponding region may be the transform kernel corresponding tothe kernel index ‘0’. In addition, the horizontal transform kernel ofthe corresponding region may be the transform kernel corresponding tothe kernel index ‘0’.

In addition, FIG. 14(b) represents the transform kernels correspondingto the kernel indexes. Referring to FIG. 14(b), a transform kernelcorresponding to a first kernel index “0” may represent the DCT-II-basedtransform kernel. A transform kernel corresponding to a second kernelindex “1” may represent the DST-VII-based transform kernel. In addition,a transform kernel corresponding to a third kernel index ‘2’ mayrepresent the DCT-VIII-based transform kernel.

According to an additional embodiment, the embodiments described abovemay be limitedly applied only to a luma block. In this case, a presetbasic transform kernel may be used for the chroma component. Forexample, the basic transform kernel may be the DCT-II-based transformkernel. In addition, the set index described above may be signaled inunits of transform blocks.

In addition, information indicating whether or not to signal the setindex may be signaled through a header of a high level including thecurrent block. Here, the high level may represent a slice/tile, apicture, or a sequence including the current block. In addition,information indicating whether or not to signal the set index may besignaled through individual flags for each prediction method. Forexample, in determining a transform kernel of a block, a flag indicatingwhether or not a set index is used may be independently set for each ofthe intra-predicted block and the inter-predicted block.

According to an embodiment, if information representing whether or notthe set index of the high level including a transform block is signaledrepresents that the set index is not explicitly signaled, the transformkernel may be determined in a different manner for the correspondingtransform block. For example, a transform kernel to be applied toinverse transform of the current transform block may be determined onthe basis of information induced from information other than the setindex. Specifically, the decoder may induce information indicating atransform kernel for the current transform block from informationsignaled in relation to the current transform block. That is,information representing the vertical transform kernel and informationindicating the horizontal transform kernel to be used for the transformblock may be implicitly signaled. According to another embodiment, ifinformation indicating whether or not the set index of the high levelincluding the transform block is signaled indicates that the set indexis explicitly signaled, the decoder may obtain a transform kernel for atransform block on the basis of the signaled set index.

According to an additional embodiment, the encoder may determine whetheror not the set index is signaled according to the number of non-zerotransform coefficients in the transform region. In this case, thedecoder may determine a transform kernel by comparing the number ofnon-zero transform coefficients in the transform region with a presetnumber. For example, if the number of non-zero transform coefficients istwo or less, the set index may not be signaled. In this case, theinverse transformation unit may perform transform on the correspondingregion using a transform kernel preset. The preset transform kernel maybe the DST-VII-based transform kernel.

FIG. 15 is a diagram representing a method of obtainingtransform-related information according to an embodiment of the presentinvention. Transform of the current block and inverse transform of thetransform coefficient of the residual signal may be performed for eachtransform block (or transform unit). The current block may be configuredwith at least one transform block. That is, the transform block may bethe current block itself or may be a block split from the current block.If the transform block is the current block itself, the size of thetransform block may be the same as the size of the current block.According to an embodiment, the current block may be split into aplurality of transform blocks on the basis of the size of the currentblock. For example, if the size of the current block is greater than themaximum transform size, the current block may be split into theplurality of transform blocks. If the size of the current block is lessthan or equal to the maximum transform size, the transform block may bethe current block itself. As an embodiment, the size of the transformblock may be the same as the size of the coding block. That is, thetransform block may be split into the same tree structure as the codingblock. However, if the size of the coding block is greater than themaximum transform size, the coding block may be split into a pluralityof transform blocks having the maximum transform size.

In an embodiment, the maximum transform size may be a value representinga length of one side. For example, the maximum transform size may be64-length (or point). In this case, if the size (width×height) of thecurrent block is (128×32), the current block may be split into twotransform blocks each having a (64×32) size. If the maximum transformsize is 32 and the size of the current block is 32×32, the transformblock is not split from the current block and may have the same size asthe current block.

If both the width and height of the current block are less than or equalto the maximum transform size, the current block is not split into aplurality of transform blocks. Accordingly, the top-left coordinate ofthe current block and the top-left coordinate of the correspondingtransform block are the same. On the other hand, if at least one of thewidth and height of the current block is greater than the maximumtransform size, the current block may be split into a plurality oftransform blocks. Accordingly, the top-left coordinate of the currentblock and the top-left coordinate of the corresponding transform blocksmay be different.

FIG. 15(a) represents a transform unit syntax. The transform unit syntaxrepresents a process related to transform and inverse transform. Thetransform unit syntax receives, as factors, (x0, y0), which is thetop-left coordinate of the transform block, tbWidth and tbHeightrepresenting the size in the horizontal and vertical directions of thetransform block, and treeType representing a tree type.

According to an embodiment, the transform-related information mayinclude coded block flag (cbf) information tu_cbf_luma, tu_cbf_cb, andtu_cdf_cr indicating whether the transform block includes at least onenon-zero transform coefficient. The cbf information may be obtainedseparately according to a component of the transform unit. For example,the transform-related information may include at least one of cbfinformation tu_cbf_luma[x0][y0] for the luma component and cbfinformation tu_cbf_cb[x0] [y0] and tu_cbf_cr[x0][y0] for two respectivechroma components.

According to an embodiment, if cbf information of the transform blockrepresents that the transform block includes at least one non-zerotransform coefficient, inverse quantization and inverse transform may beperformed on the corresponding transform block. In this case, thedecoder may perform inverse quantization on the quantized transformcoefficient using residual coding-related information. In addition, thedecoder may obtain a residual signal corresponding to the transformblock by performing inverse quantization on the inverse quantizedtransform coefficient. In contrast, if the cbf information of thecorresponding transform block represents that the transform block doesnot include at least one non-zero coefficient, the decoder may notperform inverse quantization and inverse transform on the correspondingtransform block. In this case, the decoder may obtain the residualsignal without performing inverse quantization and inverse transformprocesses for the transform block. As an example, if cbf information ofa transform block does not exist, the cbf information may be regarded asthat the transform block does not include at least one non-zerocoefficient. The encoder may perform entropy coding on cbf informationthrough the entropy coding unit described above. In addition, thedecoder may obtain cbf information from a bitstream through the entropydecoding unit described above.

A tree structure for the luma component and a tree structure for thechroma component may be encoded in the same or different ways.Accordingly, cbf information may be obtained on the basis of information(treeType) representing the tree type of the transform block (S1501).According to an embodiment, if the treeType of the transform blockrepresents SINGLE_TREE, the tree structure of the transform block forthe luma component of the current block and the tree structure of thetransform block for the chroma component of the current block may be thesame. In this case, cbf information for the luma component and cbfinformation for each of the chroma components may be obtained. Accordingto another embodiment, if the treeType of the transform block representsDUAL_TREE_LUMA or DUAL_TREE_CHROMA, the luma component and the chromacomponent of the current block may be split into transform blocks havingdifferent structures, respectively. If the treeType of the transformblock represents DUAL_TREE_LUMA, it means that the transform blockcurrently being processed is a luma component block. Accordingly, cbfinformation on the luma component may be obtained. In addition, if thetreeType information of the transform block represents DUAL_TREE_CHROMA,it means that the transform block currently being processed is a chromacomponent block. Accordingly, cbf information on the chroma componentmay be obtained.

In FIG. 15(a), tu_cbf_luma[x0][y0] is an element indicating whether theluma component transform block includes at least one non-zerocoefficient, and if tu_cbf_luma[x0][y0] is 1, it expresses that anon-zero coefficient exists in a corresponding transform block whoseposition of the top-left luma component sample of the block is (x0, y0)with the position of the top-left luma component sample of the pictureas a reference. As an example, if tu_cbf_luma[x0][y0] does not exist, itmay be inferred as 0. If tu_cbf_luma[x0][y0] is 0, it represents thatall coefficients in the corresponding luma transform block are 0, andthus the decoder may reconstruct the corresponding residual signal forthe luma component without having been subjected to additional parsing,inverse quantization, and inverse transform. If tu_cbf_luma[x0] [y0] is1, it means that one or more non-zero transform coefficients exist inthe corresponding luma transform block, and thus the parsing and inversequantization processes of a syntax element related to the transformcoefficient are required in a residual_coding syntax structure, and aresidual signal may be reconstructed by applying inverse transform tothe inverse quantized coefficient. The tu_cbf_luma[x0][y0] may beencoded/decoded using a regular coding engine of CABAC that considerscontext.

The tu_cbf_cb[x0][y0] is an element indicating whether the chromacomponent Cb transform block includes one or more non-zero coefficients,and if tu_cbf_cb[x0][y0] is 1, it expresses that a non-zero coefficientexists in a corresponding transform block whose top-left position is(x0, y0). As an example, if tu_cbf_cb[x0][y0] does not exist, it may beinferred as 0. If tu_cbf_cb[x0][y0] is 0, it represents that allcoefficients in the corresponding Cb transform block are 0, and thus thedecoder may reconstruct the residual signal for a Cb signal withouthaving been subjected to additional parsing, inverse quantization, andinverse transform. If tu_cbf_cb[x0][y0] is 1, it means that one or morenon-zero transform coefficients exist in the corresponding Cb transformblock, and thus the parsing and inverse quantization processes of thesyntax element related to the transform coefficient are required in theresidual_coding syntax structure, and a residual signal may bereconstructed by applying inverse transform to the inverse quantizedcoefficient. The tu_cbf_cb[x0][y0] may be encoded/decoded using theregular coding engine of CABAC that considers context.

The tu_cbf_cr[x0][y0] is an element indicating whether the chromacomponent Cr transform block includes one or more non-zero coefficients,and if tu_cbf_cr[x0][y0] is 1, it expresses that a non-zero coefficientexists in a corresponding transform block whose top-left position is(x0, y0). As an example, if tu_cbf_cr[x0][y0] does not exist, it may beinferred as 0. If tu_cbf_cr[x0][y0] is 0, it represents that allcoefficients in the corresponding Cr transform block are 0, and thus thedecoder may reconstruct the residual signal for a Cr signal withouthaving been subjected to additional parsing, inverse quantization, andinverse transform. If tu_cbf_cr[x0][y0] is 1, it means that one or morenon-zero transform coefficients exist in the corresponding Cr transformblock, and thus the parsing and inverse quantization processes of thesyntax element related to the transform coefficient are required in theresidual_coding syntax structure, and a residual signal may bereconstructed by applying inverse transform to the inverse quantizedcoefficient. The tu_cbf_cr[x0] [y0] may be encoded/decoded using theregular coding engine of CABAC that considers context.

Next, the encoder and decoder may check a condition for parsing MTSinformation cu_mts_flag representing whether the MTS method is used forthe current transform block (S1502). In the following, the condition forparsing cu_mts_flag will be described. First, whether or not cu_mts_flagis parsed may be determined on the basis of high level MTS informationsps_mts_intra_enabled_flag and sps_mts_inter_enabled_flag signaledthrough the high level header of the current transform block. The highlevel MTS information may be information representing whether atransform kernel other than the basic transform kernel may be used.According to an embodiment, if sps_mts_intra_enabled_flag is 0, it mayrepresent that cu_mts_flag corresponding to an intra-predicted blockbelonging to a corresponding SPS does not exist. In this case,cu_mts_flag is not parsed. In contrast, if sps_mts_intra_enabled_flag is1, it may represent that cu_mts_flag corresponding to theintra-predicted block belonging to the corresponding SPS exists. In thiscase, cu_mts_flag may be signaled and parsed.

Similarly, if sps_mts_inter_enabled_flag is 0, it may represent thatcu_mts_flag corresponding to the inter-predicted block belonging to thecorresponding SPS does not exist. In this case, cu_mts_flag is notparsed. In contrast, if sps_mts_inter_enabled_flag is 1, it mayrepresent that cu_mts_flag corresponding to the inter-predicted blockbelonging to the corresponding SPS exists. In this case, cu_mts_flag maybe signaled and parsed. The high level MTS information may be signaledthrough a high level syntax such as a header of a sequence, picture, orslice/tile.

In addition, CuPredMode[x0][y0] is a variable representing a predictionmode of a transform block whose position of the top-left luma componentsample is (x0, y0) with the luma component top-left coordinate of thepicture as a reference. If CuPredMode[x0][y0] is MODE_INTRA, itrepresents that intra prediction is applied to the corresponding block.In addition, CuPredMode[x0][y0] is MODE_INTER, it represents that interprediction is applied to the corresponding block. That is, in S1502,(CuPredMode[x0][y0] MODE_INTRA) && sps_mts_intra_enabled_flag) checks,if intra prediction is applied to the current transform block, whetherMTS may be applied. In addition, in S1502, (CuPredMode[x0][y0]MODE_INTER) && sps_mts_inter_enabled_flag) checks, if inter predictionis applied to the current transform block, whether MTS may be applied.

In addition, whether or not to parse cu_mts_flag may be determined onthe basis of tu_cbf_luma[x0][y0]. This is because if tu_cbf_luma[x0][y0]represents that all coefficients of the luma component of thecorresponding transform block are 0, it is not necessary to performinverse transform. According to an additional embodiment, only aDCT-II-based transform kernel may be used for transform for a chromacomponent. In addition, whether or not to parse cu_mts_flag may bedetermined on the basis of whether the height tbHeight and width tbWidthof the transform block are less than or equal to a preset length. Theconditions described above are summarized as follows.

Condition i) sps_mts_intra_enabled_flag==1 andCuPredMode[x0][y0]==MODE_INTRA or sps_mts_inter_enabled_flag==1 andCuPredMode[x0][y0]==MODE_INTER,

Condition ii) tu_cbf_luma[x0][y0] 1,

Condition iii) treeType !=DUAL_TREE_CHROMA,

Condition iv) tbWidth<=32, and tbHeight<=32.

If all four the conditions described above are satisfied, the decodermay parse cu_mts_flag. In addition, the encoder may perform entropycoding on cu_mts_flag and signal cu_mts_flag. If cu_mts_flag[x0][y0] is1, it may mean that the MTS method is applied to the current transformblock. In contrast, if cu_mts_flag[x0][y0] is 0, it may represent thatthe MTS method is not applied to the current transform block. In thiscase, inverse transform may be performed using a basic kernel such as aDCT-II-based transform kernel. If cu_mts_flag[x0][y0] does not exist,cu_mts_flag[x0][y0] may be inferred as ‘0’.

According to an embodiment of the present invention, a transform kernelused for transform of a transform block may be signaled stepwiselythrough the high level header of the current block and a header of thetransform block. First, information representing whether a transformkernel other than the basic transform kernel may be used in a sequence,picture, or slice/tile including the current block may be signaledthrough the high level header. In addition, the information may beindividually signaled for intra prediction and inter prediction.According to a specific embodiment, sps_mts_intra_enabled_flag andsps_mts_inter_enabled_flag may be signaled through the header of thesequence, picture, or slice/tile. The sps_mts_intra_enabled_flag may beinformation representing whether the transform kernel other than thebasic transform kernel may be used for transform of an intra predictedblock. In addition, sps_mts_inter_enabled_flag may be informationrepresenting whether the transform kernel other than the basic transformkernel may be used for transform of the inter-predicted block. The basictransform kernel may be a DCT-II based transform kernel.

That is, if sps_mts_intra_enabled_flag represents that use of atransform kernel other than the basic transform kernel is not allowedand the current block is an intra-predicted block, information relatedto the transform kernel of the transform block obtained from the currentblock may not be additionally parsed. In this case, the correspondingtransform block may be transformed on the basis of the basic transformkernel. In addition, if sps_mts_inter_enabled_flag represents that useof the transform kernel other than the default transform kernel is notallowed and the current block is an inter-predicted block, informationrelated to the transform kernel of the transform block obtained from thecurrent block may not be additionally parsed. In this case, thecorresponding transform block may be transformed on the basis of thebasic transform kernel.

On the other hand, if the information signaled through the high levelheader represents that the transform kernel other than the basictransform kernel may be used, additional information related to thetransform kernel of the corresponding transform block may be signaledthrough the header of the transform block. For example, in transforminga transform block, information representing whether the transform kernelother than the basic transform kernel is used may be signaled through aheader of the corresponding transform block.

According to an embodiment of the present invention, use of thetransform kernel other than the basic transform kernel for the transformblock may be limited according to the size of the correspondingtransform block. For example, if the size of the transform block isgreater than a preset size, use of the transform kernel other than thebasic transform kernel may be limited in transforming the correspondingtransform block. According to an embodiment, if at least one of theheight or width of a transform block is longer than a preset length, useof the transform kernel other than a basic transform kernel may not beallowed in transforming the corresponding transform block. For example,the preset length may be 32. Referring to FIG. 15(a), if a transformblock has a width of 32 or less and a height of 32 or less,cu_mts_flag[x0][y0] signaled through the header of the transform blockdescribed above may be parsed.

As shown in FIG. 15(a), if the cbf information described aboverepresents that the transform block includes at least one non-zerotransform coefficient, a process of obtaining a quantized transformcoefficient for the corresponding transform block may be performed(S1503 to S1505). In S1503, if tu_cbf_luma[x0][y0] 1, residual_codingmay be performed on the luma component of the transform block. That is,a residual_coding syntax structure residual_coding( ) process ofreceiving a specific input factor may be performed. Specifically,residual_coding( ) may receive, as input factors, the coordinates (x0,y0) of the top-left sample of a transform block, a value log 2(tbWidth)obtained by taking a logarithm of base 2 to a width of the transformblock, and a value log 2(tbHeight) obtained by taking a logarithm ofbase 2 to a height of the transform block. In addition, residual_coding() may receive a color index cdIdx as an input factor. If a value ofcdIdx is ‘0’, it means the luma component luma, if the value of cdIdx is‘1’, it means the first chroma component Cb, and if the value of cdId is‘2’, it represents the second chroma component Cr.

In S1504, if tu_cbf_cb[x0][y0]==1, residual_coding for the first chromacomponent may be performed. Specifically, residual_coding( ) may receivethe coordinates (x0, y0) of the top-left sample of the transform blockas an input factor. If a color format YCbCr is 4:2:0, the size of thetransform block for the chroma component may be half the size of theluma component block. Accordingly, residual_coding( ) may receive, asinput factors, a value log 2(tbWidth/2) obtained by taking a logarithmof base 2 to the width tbWidth/2 of the chroma component block and avalue log 2(tbHeight/2) obtained by taking a logarithm of base 2 to theheight tbHeight/2 of the chroma component block. Meanwhile, if the colorformat YCbCr is 4:4:4, the size of the chroma component block of thetransform block may be the same as the size of the luma component block.In this case, residual_coding( ) similar to residual_coding for the lumacomponent, may receive, as input factors, the value log 2(tbWidth)obtained by taking a logarithm of base 2 to the width of the transformblock, and the value log 2(tbHeight) obtained by taking a logarithm ofbase 2 to the height of the transform block.

In S1505, if tu_cbf_cr[x0] [y0]==1, residual_coding for the secondchroma component may be performed. Specifically, residual_coding( ) mayreceive the coordinates (x0, y0) of the top-left sample of the transformblock as an input factor. If the color format YCbCr is 4:2:0, the sizeof the transform block for the chroma component may be half the size ofthe luma component block. Accordingly, residual_coding( ) may receive,as input factors, the value log 2(tbWidth/2) obtained by taking alogarithm of base 2 to the width tbWidth/2 of the chroma component blockand the value log 2(tbHeight/2) obtained by taking a logarithm of base 2to the height tbHeight/2 of the chroma component block. Meanwhile, ifthe color format YCbCr is 4:4:4, the size of the chroma component blockof the transform block may be the same as the size of the luma componentblock. In this case, residual_coding( ), similar to residual_coding forthe luma component, may receive, as input factors, the value log2(tbWidth) obtained by taking a logarithm of base 2 to the width of thetransform block, and the value log 2(tbHeight) obtained by taking alogarithm of base 2 to the height of the transform block.

The decoder may perform inverse quantization and inverse transform for acorresponding transform block on the basis of the transform-relatedinformation described above. The decoder may perform inversequantization and inverse transform to obtain a residual signalcorresponding to a transform block. Specifically, the decoder may obtaina residual signal from the quantized transform coefficient of thetransform block through the residual coding syntax residual_coding( ) ofFIG. 15(b).

The residual_coding( ) of FIG. 15(b) receives, as factors, the top-leftcoordinate of a transform block obtained from the current block, a valueobtained by taking a logarithm of base 2 to a width of the transformblock, a value obtained by taking a logarithm of base 2 to a height ofthe transform block, and a color index cIdx, with the basis of thetop-left coordinate of the picture including the current block as areference. In addition, the residual coding syntax structure may includesyntax elements related to the quantized transform coefficient existedin the transform block, and the decoder may obtain a quantized transformcoefficient block by parsing syntax elements related to the quantizedtransform coefficient included in the residual syntax structure. Theinverse transformation unit of the decoder reconstructs a residualsignal corresponding to the transform block by performing inversequantization and inverse transform on the quantized transformcoefficient block.

According to an embodiment, the decoder may determine whether or not toparse a transform skip flag transform_skip_flag of the current transformblock (S1506). Conditions for parsing transform_skip_flag are asfollows.

Condition i) If transform_skip_enabled_flag, which is a 1-bit flagincluded in the high level syntax, is 1 (on): Thetransform_skip_enabled_flag may be included in any one of a sequenceparameter set (SPS), a picture parameter set (PPS), and a slice header,and if transform_skip_enabled_flag is 1, it represents thattransform_skip_flag exists in the residual coding syntax, and iftransform_skip_enabled_flag is 0, it represents that transform_skip_flagdoes not exist in the residual_coding syntax.

Condition ii) If the luma component transform block is not (cIdx!=0) orMTS is not applied (cu_mts_flag[x0][y0] 0): Transform skip may beapplied to both the luma component and the chroma component. If MTS isapplied to the luma component transform block (cIdx==0 &&cu_mts_flag[x0][y0] 1), it is obvious to perform the transform, and thusit is not necessary to encode/decode transform_skip_flag[x0][y0][cIdx].

Condition iii) If the width of the transform block is less than or equalto 4 (log 2TbWidth<=2) and the height is less than equal to 4 (log2TbHeight<=2).

Whether or not to parse transform_skip_flag[x0][y0][cIdx] may bedetermined according to the conditions described above. Iftransform_skip_flag[x0][y0][cIdx] is 1, the decoder may not perform thetransform. If transform_skip_flag[x0][y0][cIdx] is 0, the decoder maydetermine whether or not to transform the current transform block byreferring to other syntax elements. Through the residual_coding( )portion omitted in FIG. 15(b), the decoder may perform a process ofparsing and reconstructing the quantized transform coefficient.

Next, the decoder may determine whether or not to parse a set indexmts_idx (S1507). Conditions for parsing mts_idx are as follows.

i) If cu_mts_flag[x0][y0] 1 and luma component transform block (cIdx==0)

ii) If transform_skip_flag[x0][y0] 0: transform_skip_flag[x0][y0] 1, itindicates that the transform skip is applied, and thus it is notnecessary to determine which transform kernel is used.

iii) If the current transform block is at least a part of anintra-predicted block (CuPredMode[x0][y0] MODE_INTRA) and there are 3 ormore non-zero coefficients in the corresponding block (numSigCoeff>2),or if the current transform block is at least a part of aninter-predicted block ((CuPredMode[x0][y0]==MODE_INTER) (where,numSigCoeff variable represents the number of non-zero coefficients thatexist in the transform block).

Regarding the condition iii), if the number of non-zero coefficientsthat exist in the transform block is 2 or less (1 or 2), the transformusing a preset transform kernel may be performed withoutsignaling/parsing of mts_idx[x0][y0] for the current transform block. Inthis case, the preset transform kernel may be a DST-VII-based transformkernel, and a DST-IV-based transform kernel may be used instead of theDST-VII-based transform kernel.

According to an embodiment of the present invention, the energydistribution of the residual signal is correlated with a shape of theblock, and thus if the transform kernel is adaptively applied inconsideration of the shape of the block, the residual signal may beencoded more efficiently.

FIG. 16 is a diagram for describing a method of selecting a transformkernel on the basis of a shape of a block according to an embodiment ofthe present invention. According to an embodiment of the presentinvention, if the MTS flag (e.g., cu_mts_flag) indicating that MTS isapplied to the current block is 0, the encoder/decoder may select atransform kernel (or transform type) for the intra block according tothe shape of the block (or the width/height of the block). If the MTSflag is 0, an MTS index (e.g., mts_idx) indicating a specific transformkernel applied to the current block is not signaled and may be inferredas −1. In embodiments to be described later, if the MTS flag is 0, theencoder/decoder may select a transform kernel used for theintra-predicted block according to the size or shape of the currentblock, and apply DCT-II to the horizontal and vertical directions forthe inter-predicted block regardless of the shape of the block.

In the present invention, a case in which MTS is applied and a case inwhich MTS is not applied are described separately on the basis of theMTS flag, but the present invention is not limited to this expression.For example, whether or not to apply MTS may represent the same meaningas whether or not a transform kernel other than a specific predefinedtransform kernel (which may be referred to as a basic transform type, adefault transform type, etc.) is used. If MTS is applied, a transformtype (e.g., any one of a plurality of transform types or a combinedtransform type of two or more of the transform types) other than thebasic transform type may be used for the transform, and if MTS is notapplied, the basic transform type may be used for the transform. As anexample, the basic transform type may be set (or defined) as DCT-II.

In an embodiment of the present invention, if the intra-predicted blockis a quadrate block (i.e., a square block), the encoder/decoder mayapply DCT-II to a transverse direction (or a horizontal direction) and alongitudinal direction (or a vertical direction). In addition, if theintra-predicted block is a rectangular block (i.e., a non-square block),the encoder/decoder may apply DCT-II for a long-length direction andDST-VII for a short-length direction. In this case, if the short-lengthis greater than the maximum transform size of DST-VII, theencoder/decoder may use DCT-II. For example, if the maximum transformsize of DST-VII is 16-point (or 32-length), DCT-II is applied for thetransverse direction of 64×16 blocks and DST-VII is applied for thelongitudinal direction. In addition, for a 64×32 block, DCT-II may beapplied to both the transverse and longitudinal directions.

In addition, in an embodiment of the present invention, if theintra-predicted block is a square block, the encoder/decoder may applyDCT-II to the transverse and longitudinal directions, and if theintra-predicted block is a rectangular block, the encoder/decoder mayapply DCT-II to a long-length direction and use DCT-II or DST-VII for ashort-length direction by signaling a 1-bit flag. However, if the lengthof the short side is greater than the maximum transform size of DST-VII,the encoder/decoder may use DCT-II without signaling of the 1-bit flag.For example, if the maximum transform size of DST-VII is 16-point,DCT-II may be applied to the horizontal direction of 64×16, DCT-II orDST-VII may be applied to the vertical direction by parsing the 1-bitflag, and DCT-II may be applied to both horizontal and verticaldirections. DST-VII may be considered efficient due to thecharacteristics of intra prediction, but the case of using DCT-II ratherthan DST-VII may incur lower the rate-distortion cost, and thus, unlikethe previous embodiment, in this embodiment, DCT-II may be additionallyconsidered for the short-length direction with 1-bit signaling. That is,if the 1-bit flag is 0, it may represent that DCT-II is applied to theshort-length direction, and if the 1-bit flag is 1, it may representthat DST-VII is applied to the short-length direction. In oneembodiment, the 1-bit flag may be encoded/decoded using the regularcoding engine of CABAC using context.

In addition, in an embodiment of the present invention, for theintra-predicted block, the encoder/decoder may apply, in the case of thesquare block, DCT-II to the transverse and longitudinal directions andmay apply, in the case of the rectangular block, DCT-II to a long-lengthdirection and DST-IV or DST-VII to a short-length direction. However, ifthe short-length is greater than the maximum transform size of DST-VII,DCT-II may be applied thereto. As described above, DST-VII has a highimplementation complexity, and thus DST-IV having a similarlow-frequency basis function pattern may be used instead. However, ifDST-VII is replaced with DST-IV for all sizes, encoding efficiency maybe degraded, and thus DST-IV may be used for a 4-point size, which is asmall size, and DST-VII may be used for other sizes.

In addition, in an embodiment of the present invention, for theintra-predicted block, the encoder/decoder may apply, in the case of thesquare block, DCT-II to the transverse and longitudinal directions, andmay apply, in the case of the rectangular block, DCT-II to a long-lengthdirection and determine a transform kernel according to an intraprediction mode for a short-length direction. As an example, if theintra prediction mode is a DC mode or a planar mode that arenon-directional modes, the encoder/decoder may apply DCT-II to ashort-length direction.

If the intra prediction mode is a horizontal direction mode (i.e., anintra prediction mode greater than or equal to a horizontal diagonalmode and less than or equal to a center diagonal mode) for a horizontalrectangle having a transverse length longer than a longitudinal lengthof the block, the encoder/decoder may apply DCT-II to the short-lengthdirection (longitudinal direction). If the intra prediction mode is avertical direction mode (i.e., an intra prediction mode that is greaterthan the center diagonal mode and less than or equal to the verticaldiagonal mode), the encoder/decoder may apply DST-VII to theshort-length direction. If the intra prediction mode is a horizontaldirection mode, for a vertical rectangle having a longitudinal lengthlonger of a block than a transverse length thereof, the encoder/decodermay apply DST-VII to a shorter-length direction (transverse direction),and if the intra prediction mode is a vertical direction mode, theencoder/decoder may apply DST-VII.

According to an embodiment of the present invention, whether or not toapply shape adaptive transform (SAT) representing a method of adaptivelydetermining a transform kernel based on the block shape of theembodiments described above may be signaled from the encoder to thedecoder through a high level syntax. As an embodiment, whether or not toapply SAT may be signaled through a sequence parameter set (SPS), apicture parameter set (PPS), a slice header, and a tile header. Inaddition, a syntax element indicating whether or not to apply SAT may bea 1-bit flag. In the following, methods of indicating whether or not toapply SAT in a high level syntax will be described. In the presentinvention, for convenience of description, a method of adaptivelydetermining a transform kernel on the basis of a size or shape of ablock is referred to as SAT, but the present invention is not limited tothis name. For example, the SAT may be referred to as implicit MTS.

In the following, with reference to Tables 1 to 8, a syntax signalingmethod related to whether or not to apply SAT will be described as anexample. The embodiment will be described mainly on the case where aflag representing whether or not to apply SAT is included in the SPS,but the present invention is not limited thereto.

TABLE 1 sps_mts_enabled_flag sps_sat_enabled_flag

Referring to Table 1, in an embodiment of the present invention, asyntax element representing whether or not to apply SAT may be signaledindependently from a syntax element representing whether or not to applyMTS. That is, as an embodiment, sps_mts_enabled_flag indicating whetherMTS may be applied to the current sequence may be signaled, andsps_sat_enabled_flag indicating whether SAT may be applied to thecurrent sequence may be signaled regardless of a value ofsps_mts_enabled_flag. That is, apart from sps_mts_enabled_flag, it maybe determined whether SAT may be applied to the current sequenceaccording to a value of sps_sat_enabled_flag.

TABLE 2 sps_mts_enabled_flag if( !sps_mts_enabled_flag ) sps_sat_enabled_flag

Referring to Table 2, in an embodiment of the present invention, asyntax element representing whether or not to apply SAT may bedetermined according to a syntax element representing whether or not toapply MTS. As an embodiment, sps_mts_enabled_flag indicating whether MTSmay be applied to the current sequence may be signaled first, andwhether or not to signal sps_sat_enabled_flag indicating whether SAT maybe applied to the current sequence may be determined according to avalue of sps_mts_enabled_flag. For example, if sps_mts_enabled_flag is1, sps_sat_enabled_flag may not be signaled. In this case,sps_sat_enabled_flag may be set to (or inferred as) 0 without signaling,which may represent that SAT is not used for all blocks in the currentsequence. If sps_mts_enabled_flag is 0, sps_sat_enabled_flag may besignaled, and the decoder may determine whether SAT may be used in thecurrent sequence depending on a value of sps_sat_enabled_flag signaled.

TABLE 3 sps_mts_enabled_flag if( !sps_mts_enabled_flag ) sps_sat_enabled_flag

Referring to Table 3, in an embodiment of the present invention, asyntax element representing whether or not to apply SAT may bedetermined according to a syntax element representing whether or not toapply MTS. As an embodiment, sps_mts_enabled_flag indicating whether MTSmay be applied to the current sequence may be signaled first, andwhether or not to signal sps_sat_enabled_flag indicating whether SAT maybe applied to the current sequence may be determined according to avalue of sps_mts_enabled_flag. For example, if sps_mts_enabled_flag is1, sps_sat_enabled_flag may not be signaled. In this case,sps_sat_enabled_flag may be set to 1 (or inferred) without signaling,which may represent that SAT may be used for all blocks in the currentsequence. If sps_mts_enabled_flag is 0, sps_sat_enabled_flag may besignaled, and the decoder may determine whether SAT may be used in thecurrent sequence depending on a value of sps_sat_enabled_flag signaled.

TABLE 4 sps_mts_enabled_flag if( !sps_mts_enabled_flag ) sps_only_dct2_enabled_flag

Referring to Table 4, in an embodiment of the present invention, whetheror not to apply SAT may be determined according to a flag (or syntaxelement) indicating that only the DCT-II transform kernel is used in thecurrent sequence. As an embodiment, sps_mts_enabled_flag indicatingwhether MTS may be applied to the current sequence may be signaledfirst, and sps_only_dct2_enabled_flag, which is a flag indicatingwhether only the DCT-II transform kernel is used in the currentsequence, may be signaled according to sps_mts_enabled_flag. Forexample, if sps_mts_enabled_flag is 0, sps_only_dct2_enabled_flag may besignaled. If sps_only_dct2_enabled_flag is 1, it may represent that allblocks in the current sequence may not use SAT, and ifsps_only_dct2_enabled_flag is 0, it may represent that SAT may be usedin the current sequence. If sps_mts_enabled_flag is 1,sps_only_dct2_enabled_flag may be set to (or inferred as) 0 withoutsignaling, and if sps_only_dct2_enabled_flag is set to (or inferred as)0, it may represent that SAT may be used in the current sequence.

TABLE 5 sps_only_dct2_enabled_flag if( !sps_only_dct2_enabled_flag ) sps_mts_enabled_flag

Referring to Table 5, in an embodiment of the present invention, whetheror not to apply SAT may be determined according to a flag (or syntaxelement) indicating that only the DCT-II transform kernel is used in thecurrent sequence. As an embodiment, sps_only_dct2_enabled_flag, which isa flag indicating whether only the DCT-II transform kernel is used inthe current sequence, may be signaled first. In addition, whether or notto signal sps_mts_enabled_flag, which is a flag indicating whether MTSmay be used in the current sequence, may be determined according tosps_only_dct2_enabled_flag. For example, if sps_only_dct2_enabled_flagis 1, sps_mts_enabled_flag may be set to 0 (or inferred_) withoutsignaling, and sps_sat_enabled_flag, which is a flag indicating whetherSAT may be applied to the current sequence, may be set to 0 withoutsignaling. If sps_only_dct2_enabled_flag is 0, sps_mts_enabled_flag maybe signaled, and accordingly, sps_sat_enabled_flag is set to 1 withoutsignaling, and thus the decoder may apply SAT to the current sequence.

TABLE 6 sps_only_dct2_enabled_flag if( !sps_only_dct2_enabled_flag) {sps_mts_enabled_flag sps_sat_enabled_flag }

Referring to Table 6, in an embodiment of the present invention, whetheror not to apply SAT may be determined according to a flag (or syntaxelement) indicating that only the DCT-II transform kernel is used in thecurrent sequence. As an embodiment, sps_only_dct2_enabled_flag, which isa flag indicating whether only the DCT-II transform kernel is used inthe current sequence, may be signaled first. In addition, whether or notto signal sps_mts_enabled_flag, which is a flag indicating whether MTSmay be applied to the current sequence, may be determined according tosps_only_dct2_enabled_flag. For example, if sps_only_dct2_enabled_flagis 1, sps_mts_enabled_flag may be set to (or inferred as) 0 withoutsignaling, and sps_sat_enabled_flag, which is a flag indicating whetherSAT may be applied to the current sequence, may be set to (or inferredas) 0 without signaling. If sps_only_dct2_enabled_flag is 0,sps_mts_enabled_flag and sps_sat_enabled_flag may be signaled, andwhether SAT may be used in the current sequence may be determinedaccording to a value of signaled sps_sat_enabled_flag.

TABLE 7 sps_only_dct2_enabled_flag if( !sps_only_dct2_enabled_flag) {sps_mts_enabled_flag if( !sps_mts_enabled_flag) sps_sat_enabled_flag }

Referring to Table 7, in an embodiment of the present invention, whetheror not to apply SAT may be determined according to a flag (or syntaxelement) indicating that only the DCT-II transform kernel is used in thecurrent sequence and a flag indicating whether MTS may be applied to thecurrent sequence (or syntax element). As an embodiment,sps_only_dct2_enabled_flag, which is a flag indicating whether only theDCT-II transform kernel is used in the current sequence, may be signaledfirst. In addition, whether or not to signal sps_mts_enabled_flag, whichis a flag indicating whether MTS may be applied to the current sequence,may be determined according to a value of sps_only_dct2_enabled_flag.For example, if sps_only_dct2_enabled_flag is 1, sps_mts_enabled_flagmay be set to (or inferred as) 0 without signaling, andsps_sat_enabled_flag, which is a flag indicating whether SAT may beapplied to the current sequence, may be set to (or inferred as) 0without signaling.

In addition, if sps_only_dct2_enabled_flag is 0, sps_mts_enabled_flagmay be signaled. In addition, if sps_mts_enabled_flag is 0, it may bedetermined whether or not SAT may be used in the current sequenceaccording to the signaled sps_sat_enabled_flag value. On the other hand,if sps_mts_enabled_flag is 1, sps_sat_enabled_flag may be set to 0without signaling, which may represent that SAT may not be applied tothe current sequence.

According to an embodiment of the present invention, ifsps_mts_enabled_flag, which is a flag indicating whether MTS may beapplied to the current sequence, is 0, a syntax element indicatingwhether an explicit MTS may be used may be inferred as 0. If the syntaxelement indicating whether the explicit MTS may be used is 0, the MTSindex is not signaled and may be inferred as 0. If sps_mts_enabled_flagis 1, the syntax element indicating whether the explicit MTS may be usedmay be signaled. If the syntax element indicating whether the explicitMTS may be used is 1, an MTS index may be signaled. If the syntaxelement indicating whether the explicit MTS may be used is 0, animplicit MTS may be used.

TABLE 8 sps_only_dct2_enabled_flag if( !sps_only_dct2_enabled_flag) {sps_mts_enabled_flag if( !sps_mts_enabled_flag) sps_sat_enabled_flag }

Referring to Table 8, in an embodiment of the present invention, whetheror not to apply SAT may be determined according to a flag (or syntaxelement) indicating that only the DCT-II transform kernel is used in thecurrent sequence and a flag indicating whether MTS may be applied to thecurrent sequence (or syntax element). As an embodiment,sps_only_dct2_enabled_flag, which is a flag indicating whether only theDCT-II transform kernel is used in the current sequence, may be signaledfirst. In addition, whether or not to signal sps_mts_enabled_flag, whichis a flag indicating whether MTS may be applied to the current sequence,may be determined according to a value of sps_only_dct2_enabled_flag.For example, if sps_only_dct2_enabled_flag is 1, sps_mts_enabled_flag,which is a flag indicating whether MTS may be applied to the currentsequence, may be set to (or inferred as) 0 without signaling, andsps_sat_enabled_flag, which is a flag indicating whether SAT may beapplied to the current sequence, may be set to (or inferred as) 0without signaling.

In addition, if sps_only_dct2_enabled_flag is 0, sps_mts_enabled_flagmay be signaled. In addition, if sps_mts_enabled_flag is 0, it may bedetermined whether or not SAT may be used in the current sequenceaccording to the signaled sps_sat_enabled_flag value. On the other hand,if sps_mts_enabled_flag is 1, sps_sat_enabled_flag may be set to 1without signaling, which may represent that SAT may be applied to thecurrent sequence.

According to an embodiment of the present invention, ifsps_mts_enabled_flag, which is a flag indicating whether MTS may beapplied to the current sequence, is 0, a syntax element indicatingwhether an explicit MTS may be used may be inferred as 0. If a syntaxelement indicating whether or not the explicit MTS may be used is 0, theMTS index is not signaled and may be inferred as 0. Ifsps_mts_enabled_flag is 1, the syntax element indicating whether or notthe explicit MTS may be used may be signaled. If the syntax elementindicating whether or not the explicit MTS may be used is 1, an MTSindex may be signaled. If the syntax element indicating whether or notthe explicit MTS may be used is 0, an implicit MTS may be used.

FIG. 17 is a diagram illustrating a method of signaling and determininga transform kernel according to an embodiment of the present invention.Referring to FIG. 17, mts_idx[xTbY][yTbY][cIdx] represents a syntaxelement (or variable) indicating a transform kernel applied to atransform block whose top-left coordinate of the luma component is(xTbY, yTbY) with the top-left coordinate of the luma component of thepicture as a reference. The cIdx is a variable representing the colorcomponent, and may be set to 0 in the case of the luma component Y, 1 inthe case of the chroma component Cb, and 2 in the case of the chromacomponent Cr.

According to an embodiment of the present invention, a transform kernel(or transform type) used in MTS may be configured with {DST-VII,DCT-VIII}. Combinations of transform kernels that may be applied to thehorizontal and vertical directions of the transform block may beconfigured with a total of four combinations (e.g., {Horizontal:DST-VII, Vertical: DST-VII}, {Horizontal: DCT-VIII, Vertical: DST-VII},{Horizontal: DST-VII, Vertical: DCT-VIII}, {Horizontal: DCT-VIII,Vertical: DCT-VIII}). On the other hand, if MTS is not used, onecombination ({Horizontal: DCT-II, Vertical: DCT-II}) may be used. InFIG. 17, trTypeHor and trTypeVer are variables indicating the transformkernel (or transform type) applied to the horizontal direction and thevertical direction, respectively. As an example, 0 may represent DCT-II,1 may represent DST-VII, and 2 may represent DCT-VIII. If tu_mts_flagdoes not exist, mts_idx may be set to (or inferred as) −1 withoutsignaling. In this case, the DCT-II kernel may be applied to both thehorizontal direction and the vertical direction.

In one embodiment, mts_idx may be signaled with a 2-bit of a fixedlength. Referring to FIG. 17, if mts_idx is 0, it may be signaled as abit string 00, and in this case, DST-VII may be applied to both thehorizontal direction and the vertical direction. If mts_idx is 1, it maybe signaled as a bit string 01, and in this case, DCT-VIII may beapplied to the horizontal direction and DST-VII may be applied to thevertical direction. If mts_idx is 2, it may be signaled as a bit string10, and in this case, DST-VII may be applied to the horizontal directionand DCT-VIII may be applied to the vertical direction. If mts_idx is 3,it may be signaled as a bit string 11, and in this case, DCT-VIII may beapplied to both the horizontal direction and the vertical direction.

In the method of determining the transform kernel according to mts_idxdescribed above, an embodiment in which the transform kernel set isconfigured with {DST-VII, DCT-VIII} has been described, but the presentinvention is not limited thereto, and the transform kernel applied tothe horizontal direction and the vertical direction may be determinedaccording to mts_idx as in the following i) to v) in a transform kernelset configured with a plurality of transform kernels.

i) If mts_idx is −1: Horizontal—DCT-II, Vertical—DCT-II

ii) If mts_idx is 0: Horizontal—first candidate in transform set,Vertical—first candidate in transform set

iii) If mts_idx is 1: Horizontal—second candidate in transform set,Vertical—first candidate in transform set

iv) If mts_idx is 2: Horizontal—first candidate in transform set,Vertical—second candidate in transform set

v) If mts_idx is 3: Horizontal—second candidate in transform set,Vertical—second candidate in transform set

In the syntax structure related to transform of FIG. 15 previouslydescribed, a method of signaling MTS and transform skip (TS) usingdifferent syntax elements has been described, but, if MTS and TS aresignaled using one syntax element, syntax parsing efficiency may beimproved and the syntax structure may be expressed more concisely.Accordingly, a method of signaling MTS and TS using one syntax elementwill be described with reference to the following drawings.

FIG. 18 represents a syntax structure for obtaining information relatedto a multiple transform selection and transform skip according to anembodiment of the present invention, and FIG. 19 is a diagramillustrating the information related to the multiple transform selectionand transform skip according to an embodiment of the present invention.According to an embodiment of the present invention, MTS and TS may besignaled using one syntax element. In describing the embodiment of FIG.18, a description redundant with that of FIG. 15 previously describedwill be omitted. That is, syntax elements not mentioned in FIG. 18 mayrefer to contents described in FIG. 15.

Referring to FIG. 18, in S1801, the decoder may check (or determine)whether whether or not to parse a syntax element tu_mts_idx[x0][y0]indicating a transform kernel applied to the horizontal and verticaldirections of the current residual signal. The syntax element mayinclude information on whether or not to apply TS, whether or not toapply MTS, and information related to a transform kernel applied to thehorizontal and vertical directions. As an embodiment, tu_mts_idx[x0][y0]may be parsed if the following the conditions i), ii), iii), and iv) areall true, and may be encoded/decoded using a regular coding engine ofCABAC using context.

Condition i) If the variable MaxMtsIdx is greater than 0 (MaxMtsIdx>0)

Here, MaxMtsIdx represents a variable representing the maximum value oftu_mts_idx, and may be determined according to the method with referenceto FIG. 15(a) previously described. In FIG. 19(a), MaxMtsIdx may bedetermined according to whether MTS may be used and whether MTS or TSmay be used. The second column of FIG. 19(a) represents whether thecurrent block is encoded with inter prediction and MTS may be used forthe inter-predicted block. If the current block is encoded with interprediction (i.e. CuPredMode[x0][y0] MODE_INTER), and MTS is allowed forinter prediction in SPS (i.e., sps_mts_inter_enabled_flag==1), a valueof the second column is 1, otherwise the value may be 0.

The third column of FIG. 19(a) represents whether the current block isencoded with intra prediction and MTS may be used for theintra-predicted block. If the current block is encoded with intraprediction (i.e., CuPredMode==MODE_INTRA), and MTS is allowed for interprediction in SPS (i.e., sps_mts_intra_enabled_flag==1), a value of thethird column is 1, otherwise the value may be 0. The fourth column ofFIG. 19(a) represents whether TS may be applied to the current transformblock. If it is indicated that TS may be used in the SPS(transform_skip_enabled_flag), and both the width and height of thecurrent transform block are less than or equal to the maximum transformskip size (i.e., log 2TbWidth<=MaxTsSize && log 2TbHeight<=MaxTsSize), avalue of the fourth column is 1, otherwise the value may be 0. In thiscase, if sps transform_skip_enabled_flag is 1, a value obtained bysubtracting 2 from a value, which is obtained by taking a logarithm ofbase 2 to the maximum transform skip size, may be signaled to determinethe MaxTsSize expressing the maximum transform skip size, and theMaxTsSize may be a value between 2 and 5. The value of the variableMaxMtsIdx may be determined according to the values of the second,third, and fourth columns of FIG. 19(a) described above.

Condition ii) If the current luma component transform block includes anon-zero coefficient (tu_cbf_luma[x0][y0]==1)

Referring to the condition ii), if the current transform block does notinclude a non-zero coefficient, it is not necessary to apply MTS and TS.In this case, the decoder may reconstruct the residual signal withoutperforming inverse quantization or inverse transform.

Condition iii) If a coding tree currently being processed is not a treeincluding only chroma components (treeType!=DUAL_TREE_CHROMA)

Referring to the condition iii), in the embodiment of FIG. 15 previouslydescribed, an example in which TS may be applied to both the lumacomponent and the chroma component has been described, but TS may beapplied only to the luma component. That is, if MTS and TS may beapplied only to the luma component, it is not necessary to signaltu_mts_idx in the coding tree including only the chroma component, andthus, if the coding tree currently being processed includes the lumacomponent, i.e., if treeType is SINGLE_TREE or DUAL_TREE_LUMA, thedecoder may parse tu_mts_idx.

Condition iv) If both the width and height of the transform block areless than 32 (tbWidth<=32 && tbHeight<=32)

In one embodiment, in a case where tu_mts_idx[x0][y0] does not exist, itmay be set to 0 if MaxMtsSize is 4, and may be set to 1 if MaxMtsSize is1 or 5. In FIG. 18, since the TS and MTS are signaled using oneintegrated syntax element, a 1-bit flag indicating whether the TS isused for the current block may not be parsed in the residual_codingsyntax structure of FIG. 18(b).

Referring to FIG. 19(b), in an embodiment of the present invention, theencoder may signal MTS and TS to the decoder using one integrated syntaxelement. The decoder may determine the transform kernel on the basis ofthe syntax element. In FIG. 19(b), TrHorType and TrVerType representvariables indicating transform kernels applied to the horizontal andvertical directions, respectively. IsTrafoSkip represents a variableindicating whether the transform skip is applied to the currenttransform block. As an embodiment, values of TrHorType, TrVerType, andIsTrafoSkip described above may be determined depending on values oftu_mts_idx[x0][y0].

In one embodiment, if TrHorType (or TrVerType) is −1 and IsTrafoSkip is1, it represents that the transform skip is applied to the correspondingdirection. If TrHorType (or TrVerType) is 0 and IsTrafoSkip is 0, theDCT-II kernel may be applied to the corresponding direction. IfTrHorType (or TrVerType) is 1 and IsTrafoSkip is 0, the DST-VII kernelmay be applied to the corresponding direction. If TrHorType (orTrVerType) is 2 and IsTrafoSkip is 0, the DCT-VIII kernel may be appliedto the corresponding direction. The maximum value of tu_mts_idx may berepresented by a MaxMtsIdx variable, and the binarization method oftu_mts_idx may vary depending on a MaxMtsIdx value.

In one embodiment, if MaxMtsIdx is 1 (i.e., only TS is allowed), whetheror not TS is applied may be indicated through 1-bit signaling. Iftu_mts_idx is 0 (i.e., bit value 0), TS may be applied to the currentluma component transform block, and if tu_mts_idx is 1 (i.e., bit value1), DCT-II may be applied to both the horizontal and vertical directionsof the current luma component transform block. If MaxMtsIdx is 4 (i.e.,TS is not allowed and MTS is allowed), the first bit of the bit columnmay represent whether MTS is applied to the corresponding luma transformblock, and the second, third, and fourth bits of the bit string mayindicate a transform kernel applied if MTS is used. If tu_mts_idx is 0(e.g., bit string 0), the DCT-II kernel may be applied to both thehorizontal and vertical directions of the current transform block. Iftu_mts_idx is 1 (e.g., bit string 10), the DST-VII kernel may be appliedto both the horizontal and vertical directions of the current transformblock. If to mts_idx is 2 (e.g., bit string 110), DCT-VIII may beapplied to the horizontal direction and DST-VII may be applied to thevertical direction of the current transform block. If tu_mts_idx is 3(e.g., bit string 1110), DST-VII may be applied to the horizontaldirection and DCT-VIII may be applied to the vertical direction of thecurrent transform block. If tu_mts_idx is 4 (e.g., bit string 1111),DCT-VIII may be applied to both the horizontal and vertical directionsof the current transform block.

In one embodiment, if MaxMtsIdx is 5 (if both TS and MTS are allowed),the first bit of the bit string may represent whether TS is applied tothe corresponding luma transform block. In this case, the second bit ofthe bit string may represent whether MTS is applied to the correspondingluma transform block, and the third, fourth, and fifth bits of the bitstring may indicate the transform kernel applied if MTS is used. Iftu_mts_idx is 0 (e.g., bit string 0), TS may be applied to the currenttransform block. If to tms_idx is 1 (e.g., bit string 10), DCT-II may beapplied to both the horizontal and vertical directions of the currenttransform block. If tu_mts_idx is 2 (e.g., bit string 110), DST-VII maybe applied to both the horizontal and vertical directions of the currenttransform block. If tu_mts_idx is 3 (e.g., bit string 1110), DCT-VIIImay be applied to the horizontal direction and DST-VII may be applied tothe vertical direction of the current transform block. If tu_mts_idx is4 (e.g., bit string 11110), DST-VII may be applied to the horizontaldirection and DCT-VIII may be applied to the vertical direction of thecurrent transform block. If tu_mts_idx is 5 (e.g., bit string 11111),DCT-VIII may be applied to both the horizontal and vertical directionsof the current transform block.

In the embodiment described above, if MTS is used, an example in whichthe transform kernel to be applied is signaled using a truncated unarybinarization method using a 1-bit or a 3-bit has been described, butsignaling may also be performed by using a fixed length of 2-bit. Inaddition, in the above embodiment, a case in which the transform kernelcandidates configuring the transform kernel set are DST-VII and DCT-VIIIhas been described as an example, but the present invention is notlimited thereto, and even in a transform kernel set configured with twoarbitrary transform kernels, a transform kernel applied to thehorizontal and vertical directions may be determined according to thevalue of the bit string indicating the transform kernel as shown in thefollowing i) to v).

i) Bit string ‘0’: Horizontal—DCT-II, Vertical—DCT-II

ii) Bit string ‘10’: Horizontal—first candidate in transform set,Vertical—first candidate in transform set

iii) Bit string ‘110’: Horizontal—second candidate in transform set,Vertical—first candidate in transform set

iv) Bit string ‘1110’: Horizontal—first candidate in transform set,Vertical—second candidate in transform set

v) Bit string ‘1111’: Horizontal—second candidate in transform set,Vertical—second candidate in transform set

In the residual signal of the inter-predicted block a tendency that theresidual signal concentratedly exists at a specific position of theblock may occur. For example, a pattern in which the residual signal hasa large energy at the boundary of the residual signal block and theresidual signal has relatively small energy at the center portion andthe opposite boundary of the block may occur. If transform is performedon the residual signal of the inter-predicted block using such apattern, encoding efficiency for inter prediction may be improved. Itwill be described with reference to the following drawings.

FIG. 20 is a diagram for describing a sub-block transform (SBT) methodaccording to an embodiment of the present invention. If SBT is applied,the encoder/decoder may perform transcoding by splitting the currentcoding unit (or coding block) into a plurality of transform units (ortransform blocks). As an embodiment, the encoder/decoder may performtranscoding by splitting the current coding unit into two transformunits having a size of 1:1 or 1:3. In this case, transform is performedonly on one transform unit assuming that the residual signal isconcentrated in one of the two split transform units, and it may beassumed that there is no residual signal in the other transform unit. InFIGS. 20 (a), (b), (c), and (d), the transform unit of a region (thatis, the shaded region) indicated by A represents a region wheretransform is performed. Whether or not a non-zero coefficient exists inthe corresponding region in the transform block after quantization maybe determined through signaling of syntax elements (tu_cbf_luma,tu_cbf_cb, tu_cbf_cr) representing information of non-zero transformcoefficient described in FIGS. 15 and 18. The remaining one region otherthan A region is a transform unit region for which it is assumed thatthere is no residual signal, and the transform and inverse quantizationprocesses may be omitted, and tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr maybe all set to (or inferred as) 0 without signaling.

Referring to FIG. 20(a), if a size of a coding unit is W×H, theencoder/decoder may split the coding unit in a vertical direction tosplit the coding unit into two transform units each having a size of(W/2)×H. Alternatively, if a 1-bit flag indicating {¼, ¾} splitting is1, the coding unit may be split into two transform units having sizes of(W/4)×H and (3W/4)×H. In both cases, the transform may be performed onthe left transform unit, and DCT-VIII may be applied to the horizontaldirection and DST-VII may be applied to the vertical direction. As anexample, if the width or height of the corresponding transform unit isgreater than the maximum size of the DST-VII/DCT-VIII kernel allowed inthe SBT, DCT-II may be applied to the corresponding direction. As anexample, the maximum size of the DST-VII/DCT-VIII kernel allowed in SBTmay be 32-point (or length).

Referring to FIG. 20(b), if the size of the coding unit is W×H, theencoder/decoder may split the coding unit in a vertical direction tosplit the coding unit into two transform units each having a size of(W/2)×H. Alternatively, if a 1-bit flag indicating {¼, ¾} splitting is1, the coding unit may be split into two transform units having sizes of(W/4)×H and (3 W/4)×H. In both cases, the transform may be performed onthe transform unit on the right side, and DST-VII may be applied to boththe horizontal and vertical directions. As an example, if the width orheight of the corresponding transform unit is greater than the maximumsize of the DST-VII/DCT-VIII kernel allowed in the SBT, DCT-II may beapplied to the corresponding direction. As an example, the maximum sizeof the DST-VII/DCT-VIII kernel allowed in SBT may be 32-point.

Referring to FIG. 20(c), if the size of the coding unit is W×H, theencoder/decoder may split the coding unit in a horizontal direction tosplit the coding unit into two transform units each having a size ofW×(H/2). Alternatively, if a 1-bit flag indicating {¼, ¾} splitting is1, the coding unit may be split into two transform units having sizes ofW×(H/4) and W×(3H/4). In both cases, the transform may be performed onthe upper transform unit, and DST-VII may be applied to the horizontaldirection and DCT-VIII may be applied to the vertical direction. As anexample, if the width or height of the corresponding transform unit isgreater than the maximum size of the DST-VII/DCT-VIII kernel allowed inthe SBT, DCT-II may be applied to the corresponding direction. As anexample, the maximum size of the DST-VII/DCT-VIII kernel allowed in SBTmay be 32-point.

Referring to FIG. 20(d), if the size of the coding unit is W×H, theencoder/decoder may split the coding unit in a horizontal direction tosplit the coding unit into two transform units each having a size ofW×(H/2). Alternatively, if a 1-bit flag indicating {¼, ¾} splitting is1, the coding unit may be split into two transform units having sizes ofW×(H/4) and W×(3H/4). In both cases, transform may be performed on thelower transform unit, and DST-VII may be applied to the horizontaldirection and DCT-VIII may be applied to both the horizontal directionand the vertical direction. As an example, if the width or height of thecorresponding transform unit is greater than the maximum size of theDST-VII/DCT-VIII kernel allowed in the SBT, DCT-II may be applied to thecorresponding direction. As an example, the maximum size of theDST-VII/DCT-VIII kernel allowed in SBT may be 32-point.

In one embodiment, the SBT may be applied to the inter-predicted blockwith cu_cbf, which is a syntax element indicating whether or not asyntax structure (e.g., transform tree syntax structure) related totransform exists, of 1 and whether or not to apply SBT may be signaledat the coding unit level. If SBT is applied at the coding unit level, a1-bit flag indicating whether splitting is {¼, ¾} splitting or ½splitting may be signaled. In addition, a 1-bit flag indicating whethersplitting is horizontal splitting or vertical splitting may be signaled.In addition, a 1-bit flag indicating a transform unit in which transformis performed among the two split transform units may be additionallyindicated.

If SBT is applied, the transform kernel is determined according to theposition of the transform unit, and thus the MTS flag and/or the MTSindex may not be signaled in the transform unit syntax structure of FIG.15. As the transform kernel applied to transform units in whichtransform is performed in SBT, any one of DST-VII/DCT-VIII/DCT-II may beselected according to the position of the corresponding transform unitand/or the width and height of the corresponding transform unit, andwhether or not to apply TS may be selected in the similar way. If TS maybe used in the transform unit syntax structure of FIG. 18 previouslydescribed (that is, if the fourth the condition of FIG. 19(a) is true),MaxMtsIdx may be determined as 1, and whether TS is applied through1-bit signaling may be indicated. If TS may not be used in the transformunit syntax structure of FIG. 18 (that is, if the fourth the conditionin FIG. 19(a) is false), MaxMtsIdx may be determined as 0, and the MTSindex syntax element may not be signaled.

As the size of the transform kernel increases, the energy compactionperformance is significantly improved. Accordingly, there is apossibility that the encoding efficiency is higher in the case ofperforming transform by applying any one of DCT-II, DST-VII, andDCT-VIII rather than TS, as the size of the transform block increases,than in the case of performing transform by applying TS. Accordingly, inan embodiment, if the size of the transform block is greater than orequal to a predetermined specific size, the MTS index syntax element maynot be signaled regardless of whether the TS may be used.

In one embodiment, whether or not SBT may be used may be signaled with a1-bit flag by being included in any one of a sequence parameter set(SPS), a picture parameter set (PPS), a slice header, a tile groupheader, or a tile header, and in the present invention, the flag may bereferred to as sps_sbt_enabled_flag. In the following, with reference toTables 9 to 11, embodiments of a method of signaling a syntax related toSBT at the SPS level will be described as an example. In the embodimentof the present invention, the signaling method at the SPS level ismainly described, but the present invention is not limited thereto, andthe signaling structure described below may be equally applied ifsignaling is performed at any one level among high level syntax.

TABLE 9 sps_sbt_enabled_flag

Referring to Table 9, in one embodiment of the present invention, theencoder/decoder may determine whether the SBT may be used in the currentsequence by signaling sps_sbt_enabled_flag, which is a 1-bit flagrepresenting whether SBT may be used in the current sequence.

TABLE 10 sps_dct2_only_enabled_flag if( !sps_dct2_only_enabled_flag ) sps_sbt_enabled_flag

Referring to Table 10, in an embodiment of the present invention, theencoder/decoder may determine whether SBT may be used depending onwhether or not a kernel other than DCT-II may be used in the currentsequence. In Table 10, sps_dct2_only_enabled_flag is a flag (or syntaxelement) representing that only the DCT-II transform kernel may be usedin the current sequence. Whether or not to signal sps_sbt_enabled_flagmay be determined depending on sps_dct2_only_enabled_flag. For example,if sps_dct2_only_enabled_flag is 0, the decoder may parsesps_sbt_enabled_flag, which is a 1-bit flag indicating whether SBT maybe used in the current sequence. The decoder may determine whether SBTmay be used in the current sequence depending on thesps_sbt_enabled_flag value. If sps_dct2_only_enabled_flag is 1,sps_sbt_enabled_flag is not signaled and is set to (or inferred as) 0,and SBT may not be used in the current sequence.

TABLE 11 sps_sbt_enabled_flag if( sps_sbt_enabled_flag ) sps_sbt_dct2_enabled_flag

Referring to Table 11, in an embodiment of the present invention, theencoder/decoder may determine whether a kernel other than DCT-II may beused in the current sequence depending on whether or not the SBT may beused in the current sequence. As an embodiment, sps_sbt_enabled_flag,which is a 1-bit flag indicating whether SBT may be used in the currentsequence, may be signaled first. If sps_sbt_enabled_flag is 1,sps_sbt_dct2_enabled_flag, which is a 1-bit flag indicating that onlyDCT-II is allowed as a transform kernel used when SBT is applied, may besignaled. That is, if sps_sbt_dct2_enabled_flag is 1, when SBT isapplied, the coding unit may be split into two transform units, andDCT-II may be applied to both the horizontal and vertical direction ofthe transform unit. If sps_sbt_enabled_flag is 0,sps_sbt_dct2_enabled_flag may be set to 0.

The encoder/decoder may reconstruct a block in units of transform units,and in the case of intra prediction, a prediction block for the currentblock may be generated by using reconstructed samples around the currentblock as a reference sample. In the case of intra prediction, aprediction error tends to increase as the distance from the referencesample increases. Accordingly, if the transform unit is split into smallblocks for the current coding unit and sequentially reconstructed inunits of split small blocks, the prediction error, that is, a residualsignal, may be reduced and encoding efficiency may be improved. An intrasub-partitions (ISP) encoding mode is a method of performing predictionand reconstruction by splitting the current coding unit into multiplesplitting units, if intra prediction is selected for the current codingunit.

FIG. 21 is a diagram illustrating a method of splitting a block in anintra sub-partitions encoding mode according to an embodiment of thepresent invention. According to an embodiment of the present invention,the current coding block may be split in a horizontal or verticaldirection. As an embodiment, the number of transform units splitdepending on the size of the coding unit may vary. For example, if thesize of the coding unit is 4×4, ISP encoding mode may not be applied. Acoding unit having a size of 4×8 or 8×4 may be split into two transformunits. The remaining coding units of different sizes may be split intofour transform units. The size of the coding unit to which ISP encodingmode may be applied may be determined as a case (a condition A) wherethe width or height of a coding unit is less than or equal to themaximum transform block size of luma component and a case (a conditionB) where a product of the width and height of a coding unit is greaterthan a square of the minimum transform block size of luma component. Inthis case, the maximum transform block size of luma component may be 64,and the minimum transform block size of luma component may be 4. Thesize of the coding unit that satisfies the condition A may be classifiedinto conditions i), ii), and iii) as follows.

Condition i) If the width of the coding unit is less than or equal tothe maximum transform block size of luma component, and the height ofthe coding unit is greater than the maximum transform block size of lumacomponent.

Condition ii) If the width of the coding unit is greater than themaximum transform block size of luma component, and the height of thecoding unit is less than or equal to the maximum transform block size ofluma component.

Condition iii) If both the width and height of the coding unit are lessthan or equal to the maximum transform block size of luma component

Among the above conditions i), ii), and iii), in the case of thecondition i), the coding unit may be split into multiple transform unitswithout separate signaling, where the width of the transform unit is thesame as the width of the coding unit, and the height of the transformunit is the same as the maximum transform block size of luma component.In the case of the condition ii), the coding unit may be split intomultiple transform units without separate signaling, where the width ofthe transform unit is equal to the maximum transform block size of lumacomponent, and the height of the transform unit is equal to the heightof the coding unit. That is, in the case of the conditions i) and ii),transform block splitting may be performed using the size of the codingunit and the maximum transform block size of luma component withoutsignaling, and thus the ISP encoding mode requiring signaling may not beused. Accordingly, the condition A may be determined as a case whereboth the width and height of the coding unit are less than or equal tothe maximum transform block size of luma component.

FIG. 21(a) shows a method of splitting a transform unit that may beapplied to a 4×8 or 8×4 coding unit. As an embodiment, if horizontalsplitting is indicated for a coding unit of W×H size, the coding unitmay be split into two transform units of W×(H/2) size, and if verticalsplitting is indicated, the coding unit may be split into two transformunits of (W/2)×H size. Whether splitting is horizontal splitting orvertical splitting may be indicated by a 1-bit flag.

FIG. 21(b) shows a method of splitting a transform unit that may beapplied to a coding unit whose size is not 4×8 and 8×4. As anembodiment, if horizontal splitting is indicated for a coding unit ofW×H size, the coding unit may be split into four transform units ofW×(H/4) size, and if vertical splitting is indicated, the coding unitmay be split into four transform units of (W/4)×H size. Whethersplitting is horizontal splitting or vertical splitting may be indicatedby a 1-bit flag.

FIG. 22 is a diagram illustrating a method of selecting a transformkernel applied to a split transform unit, if the intra sub-partitions(ISP) mode according to an embodiment of the present invention isapplied. According to an embodiment of the present invention, if ISP isapplied, a transform kernel applied to each transform unit may bedetermined depending on a size of a transform block and the intraprediction mode. In FIG. 22, predModeIntra is a variable representingthe intra prediction mode used for the current coding unit, andtrTypeHor and trTypeVer are variables representing the transform kernelsapplied to the horizontal and vertical directions, respectively.Referring to FIG. 22, description is made on the assumption thatsixty-seven intra prediction modes are used, but the present inventionis not limited thereto, and the same may be applied even if thewide-angle mode previously described in FIG. 6 is additionally used, inaddition to the sixty-seven intra prediction modes.

According to an embodiment of the present invention, the intraprediction mode may be configured with two non-directional predictionmodes (INTRA_PLANAR, INTRA_DC) and sixty-five directional predictionmodes (INTRA_ANGULAR2, INTRA_ANGULAR3, . . . , INTRA_ANGULAR66), andvalues of the trTypeHor and trTypeVer variables may be determineddepending on each prediction mode. Here, trTypeHor and trTypeVerrepresent transform kernels (or transform types) applied to thehorizontal and vertical directions, respectively. If the value oftrTypeHor or trTypeVer is 0, it represents that the DCT-II transformkernel is applied to the corresponding direction, and if the trTypeHoror trTypeVer value is 1, it represents that DST-VII transform kernel isapplied to the corresponding direction.

In one embodiment, if the width of the transform block is less than theminimum transform block size of luma component or greater than themaximum size of DST-VII used in ISP, DCT-II may be applied to thehorizontal direction. In addition, if the height of the transform blockis less than the minimum transform block size of luma component orgreater than the maximum size of DST-VII used in ISP, DCT-II may beapplied to the vertical direction. As an example, the minimum transformblock size of luma component may be defined as 4, and the maximum sizeof DST-VII may be defined as 16. The minimum transform block size may bereferred to as a first threshold value, and the maximum size of DST-VIImay be referred to as a second threshold value.

In addition, in an embodiment, the encoder/decoder may apply DST-VII toboth the horizontal direction and the vertical direction for thenon-directional prediction mode. In addition, the encoder/decoder mayapply DST-VII to the horizontal direction and DCT-II to the verticaldirection for the horizontal directional mode (INTRA_ANGULAR2,INTRA_ANGULAR3, . . . , INTRA_ANGULAR32, INTRA_ANGULAR33). Theencoder/decoder may apply DCT-II to the horizontal direction and DST-VIIto the vertical direction for the vertical directional mode(INTRA_ANGULAR34, INTRA_ANGULAR35, . . . , INTRA_ANGULAR65,INTRA_ANGULAR66).

In one embodiment, if the current coding unit is coded with intraprediction, a 1-bit flag indicating whether to apply ISP at the codingunit level may be signaled. If ISP is applied to the current codingunit, a 1-bit flag indicating whether splitting is horizontal splittingor vertical splitting may be signaled. If ISP is applied to the currentblock, the transform kernel applied to each split transform unit may bedetermined without signaling depending on the intra prediction mode andthe size of the transform unit, and thus the MTS flag and/or the MTSindex may not be signaled in the transform unit syntax structure of FIG.15. In addition, TS is not used in ISP encoding mode, and thus atransform skip flag may not be signaled. In addition, the MTS index maynot be signaled in the transform unit syntax structure of FIG. 18.

As described in FIG. 22, if ISP is applied to the current block, atransform kernel applied to each transform block may be determinedwithout signaling based on the intra prediction mode. Meanwhile, thismay increase the complexity of hardware decoder implementation. In anexisting hardware decoder pipeline structure, decoding of the intraprediction mode and reconstruction of the transform coefficient may beconcurrently progressed, but if ISP mode is applied, the reconstructionof the transform coefficient may be progressed only after the decodingof the intra prediction mode is completed, and thus a delay may becaused by that amount. Accordingly, in the following, methods ofdetermining the transform kernel applied to each transform block withoutbeing based on the intra prediction mode in ISP encoding mode will bedescribed.

In one embodiment of the present invention, the encoder/decoder maydetermine a transform kernel (or transform type) of a block to which ISPmode is applied by comparing the width or height of the transform blockwith a predefined specific threshold value. The DST-VII kernel wellexpresses the characteristics of the residual signal when intraprediction is used, and thus, if the width of the transform block isless than a specific threshold value, DST-VII may be applied to thehorizontal direction, and if the width of the transform block is greaterthan the threshold value, DCT-II may be applied to the horizontaldirection. Here, the specific threshold value may be defined as the samevalue as the previously described second threshold value (i.e., themaximum size of DST-VII used in ISP). If the height of the transformblock is less than or equal to the threshold value, DST-VII may beapplied to the vertical direction, and if the height of the transformblock is greater than the threshold value, DCT-II may be applied to thevertical direction. In this case, the threshold value may be defined asan arbitrary value of 32 or less. As described above, as an embodiment,if the width of the transform block is less than the minimum transformblock size (i.e., the first threshold value) of luma component orgreater than the maximum size of DST-VII used in ISP, DCT-II may beapplied to the horizontal direction. In addition, if the height of thetransform block is less than the minimum transform block size of lumacomponent or greater than the maximum size of DST-VII used in ISP,DCT-II may be applied to the vertical direction. As an example, theminimum transform block size of luma component may be defined as 4, andthe maximum size of DST-VII may be defined as 16.

In addition, in an embodiment of the present invention, DST-VII may beapplied to both the horizontal and vertical directions of a square blockin which both the width and height of the transform block are less thanor equal to the threshold value, and DCT-II may be applied to thehorizontal direction and the vertical direction of a square block inwhich both the width and the height of the transform block are greaterthan the threshold value. In a rectangular block, if the short side isless than or equal to the threshold value, DST-VII may be applied to thecorresponding direction, and if the short side is greater than thethreshold value, DCT-II may be applied to the corresponding direction.For the long side of the rectangular block, DCT-II may be applied to thecorresponding direction.

In addition, in an embodiment of the present invention, the transformkernel applied to each transform unit may be determined by utilizingsignaling of an MTS-related syntax element described in FIGS. 15 and 18.For example, if the transform unit syntax structure of FIG. 18 is used,even if ISP is used in the current coding unit, the MTS index maysignal. If the MTS index is signaled for each transform unit, thesignaling overhead may increase. Therefore, in the processing order, theMTS index is signaled only in the first transform unit (i.e.,tu_cbf_luma==1), in which a non-zero transform coefficient exists forthe luma component transform block, in the processing order, and thesignaled MTS index may be shared by all transform units. In this case,as an example, whether or not to apply TS may be signaled together.

In one embodiment, whether or not ISP encoding mode may be used may besignaled with a 1-bit flag by being included in any one of a sequenceparameter set (SPS), a picture parameter set (PPS), a slice header, atile group header, and a tile header, and in the present invention, theflag may be referred to as sps_isp_enabled_flag. In the following, withreference to Tables 12 to 14, embodiments of a method of signaling asyntax related to ISP encoding mode at the SPS level will be describedas an example. In the embodiment of the present invention, the signalingmethod at the SPS level is mainly described, but the present inventionis not limited thereto, and if signaling is performed at any one levelamong high level syntax, the signaling structure described below may beequally applied.

TABLE 12 sps_isp_enabled_flag

Referring to Table 12, in an embodiment of the present invention, theencoder/decoder may determine whether ISP encoding mode may be used inthe current sequence by signaling sps_isp_enabled_flag, which is a 1-bitflag representing whether ISP encoding mode may be used in the currentsequence.

TABLE 13 sps_dct2_only_enabled_flag if( !sps_dct2_only_enabled_flag ) sps_isp_enabled_flag

Referring to Table 13, in an embodiment of the present invention, theencoder/decoder may determine whether ISP encoding mode may be useddepending on whether a kernel other than DCT-II may be used in thecurrent sequence. In Table 13, sps_dct2_only_enabled_flag is a flag (orsyntax element) representing that only the DCT-II transform kernel maybe used in the current sequence. Whether or not to signalsps_isp_enabled_flag, which is a flag (or syntax element) indicatingwhether ISP encoding mode may be used in the current sequence, may bedetermined depending on sps_dct2_only_enabled_flag. For example, ifsps_dct2_only_enabled_flag is 0, the decoder may parsesps_isp_enabled_flag, which is a 1-bit flag indicating whether ISPencoding mode may be used in the current sequence. The decoder maydetermine whether ISP encoding mode may be used in the current sequencedepending on the sps_isp_enabled_flag value. Ifsps_dct2_only_enabled_flag is 1, sps_isp_enabled_flag is not signaledand is set to (or inferred) 0, and ISP encoding mode may not be used inthe current sequence.

TABLE 14 sps_isp_enabled_flag if( sps_isp_enabled_flag ) sps_isp_dct2_enabled_flag

Referring to Table 14, in an embodiment of the present invention, theencoder/decoder may determine whether a kernel other than DCT-II may beused in the current sequence depending on whether or not ISP encodingmode may be used in the current sequence. As an embodiment,sps_isp_enabled_flag, which is a 1-bit flag indicating whether SBT maybe used in the current sequence, may be signaled first. Ifsps_isp_enabled_flag is 1, sps_isp_dct2_enabled_flag, which is a 1-bitflag indicating that only DCT-II is allowed, may be signaled to thetransform kernel used when ISP encoding mode is applied. That is, in thecase where sps_isp_dct2_enabled_flag is 1, when ISP encoding mode isapplied, the coding unit may be split into a plurality of transformunits, and DCT-II may be applied to both the horizontal and verticaldirections of the transform unit. If sps_isp_enabled_flag is 0,sps_isp_dct2_enabled_flag may be set to 0.

FIG. 23 shows an example of a transform unit syntax structure and anexample of a method of determining a transform type according to anembodiment to which the present invention is applied. According to anembodiment of the present invention, whether or not to apply MTS and/ora transform kernel (or transform type) may be determined depending onwhether or not to skip the transform. If transform skip is applied tothe current transform block, it is not necessary to indicate (or signal)the transform kernel applied to the current transform block. Therefore,signaling bits required for transform may be reduced and compressionefficiency may be improved by signaling whether or not to apply atransform skip before the MTS-related syntax element and signaling anMTS-related syntax element depending on whether or not to apply thetransform skip. In describing the embodiment of the present inventionwith reference to FIG. 23, the decoder is mainly described forconvenience of description, but the method of determining transform skipand MTS according to the present embodiment may be applied substantiallyequally to the encoder.

FIG. 23(a) illustrates a method of signaling/parsing transform skip andMTS information according to an embodiment of the present invention. Asan example, transform skip and MTS may be processed at the transformunit level as shown in FIG. 23(a). However, FIG. 23(a) is an example,and the present invention is not limited thereto, and a proposed methodmay be processed in substantially the same way at levels (e.g., codingunit level, prediction unit level) other than that of the transformunit. Referring to FIG. 23(a), the syntax structure of the transformunit represents a syntax structure related to a transform unit for whicha transform process is performed. The transform unit syntax may receivex0, y0, tbWidth, tbHeight, treeType, and subTuIndex as inputs. (x0, y0)may represent the position of the top-left luma sample of the currenttransform block, and may be a relative position with the position of thetop-left luma sample of the current picture as a reference. Thevariables tbWidth and tbHeight represent the width and height of thecurrent transform block, respectively.

A coding tree structure for the luma component and a coding treestructure for the chroma component may be coded with the same structureor different structures. Accordingly, the treeType variable may indicatethe type of coding tree currently being processed. If the treeType isSINGLE_TREE, it means that the luma component and the chroma componentare coded with the same coding tree structure. In this case, thetransform unit may include a luma transform block and a chroma transformblock according to a color format. If the treeType is DUAL_TREE_LUMA,when the luma component and the chroma component are coded with a codingtree structure having different structures, it may represent that thecoding tree currently being processed is a coding tree of the lumacomponent. In this case, the transform unit may include the lumatransform block. If the treeType is DUAL_TREE_CHROMA, when the lumacomponent and the chroma component are coded with a coding treestructure having different structures, it may represent that the codingtree currently being processed is a coding tree of the chroma component.In this case, the transform unit may include the chroma transform blockaccording to the color format. The subTuIndex represents an index oftransform unit currently being processed.

In the transform unit syntax structure (i.e., transform unit( ), first,the encoder may indicate whether a current transform block has one ormore non-zero transform coefficients by signaling a coded block flag CBFfor a luma block, a Cb chroma block, and a Cr chroma block. Similarly,the decoder may determine (or induce) whether the current transformblock has one or more non-zero transform coefficients by parsing thecoded block flag CBF for the luma block, Cb block (or Cb chroma block),and Cr block (or Cr chroma block). For example, if tu_cbf_luma[x0][y0]is 1, it may represent that the current luma transform block includesone or more non-zero transform coefficients, and if tu_cbf_luma[x0][y0]is 0, it may represent that all transform coefficients of the currentluma transform block are 0. If tu_cbf_cb[x0][y0] is 1, it may representthat the current Cb transform block includes one or more non-zerotransform coefficients, and if tu_cbf_cb[x0][y0] is 0, it may representthat all transform coefficients of the current Cb transform block are 0.If tu_cbf_cr[x0][y0] is 1, it may represent that the current Crtransform block includes one or more non-zero transform coefficients,and if tu_cbf_cr[x0][y0] is 0, it may represent that all transformcoefficients of the current Cr transform block are 0.

In step S2301 (i.e., the first the conditional statement), the decodermay check (or examine) a preset condition for determining whether or notto apply transform skip and/or MTS. As an example, if the following theconditions i), ii), iii), iv), and v) are all true, the decoder mayperform parsing (S2303, S2305) on transform skip and MTS information(i.e., syntax element) according to the second the conditional statementof step S2302 and the third the conditional statement of step S2304,respectively, which will be described later.

Condition i) tu_cbf_luma[x0][y0]==1: The decoder may check whether thevalue of tu_cbf_luma[x0][y0] is 1. If the value of tu_cbf_luma[x0][y0]is 0, it represents that all transform coefficients of the current lumatransform block are 0, and thus the decoder may reconstruct the residualsignal to zero without performing inverse quantization and inversetransform. Accordingly, if the value of tu_cbf_luma[x0][y0] is 0,transform skip and MTS information may not be signaled and parsed.

Condition ii) treeType !=DUAL_TREE_CHROMA: The decoder may check whetherthe tree type is a tree type including a luma transform block. Bothtransform skip and MTS may be applied only to the luma transform block,and thus if the current transform unit (or transform block) does notinclude the luma transform block, transform skip and MTS information maynot be signaled and parsed. That is, if the tree type isDUAL_TREE_CHROMA, the current transform unit does not include the lumatransform block, and thus transform skip and MTS information may not besignaled and parsed.

Condition iii) (tbWidth<=32) && (tbHeight<=32): The decoder may checkwhether the width and height of the current transform block are lessthan or equal to 32. The maximum size of the transform block to whichtransform skip and MTS may be applied is 32×32, and thus if both thewidth and height of the transform block are 32 or less, signaling andparsing of transform skip and MTS information may be performed dependingon additional the conditions.

Condition iv) IntraSubPartitionsSplitType[x0][y0]==ISP_NO_SPLIT: Thedecoder may check whether ISP is not applied to the current coding unit(or coding block). If ISP is applied to the current coding unit,transform and inverse transform may be performed according to a presetmethod. Accordingly, if ISP is applied to the current coding unit,transform skip and MTS information may not be signaled and parsed.IntraSubPartitionsSplitType represents a variable indicating a blocksplitting method by ISP. The IntraSubPartitionsSplitType may indicatethe block splitting method by ISP according to ISP information signaledat the coding unit level. If IntraSubPartitionsSplitType is notISP_NO_SPLIT, it may represent that ISP is applied to the current block.Accordingly, if IntraSubPartitionsSplitType is not ISP_NO_SPLIT,transform skip and MTS information may not be signaled and parsed.

Condition v) !cu_sbt_flag: The decoder may check whether sub-blocktransform (SBT) is not applied to the current coding unit. If SBT isapplied to the current coding unit, transform and inverse transform maybe performed according to a preset method. Accordingly, if SBT isapplied to the current coding unit, transform skip and MTS informationmay not be signaled and parsed. The cu_sbt_flag represents a syntaxelement indicating whether SBT is applied to the current coding unit.The cu_sbt_flag may be signaled and parsed at the coding unit level. Ifcu_sbt_flag is 1, it may represent that SBT is applied to the currentcoding unit, and transform skip and MTS information may not be signaledand parsed.

If the conditions i) to v) described above are all true, in step S2302,the decoder may check (or examine) the following the conditions a) andb) for determining whether or not to apply the transform skip. As anexample, if the following the conditions a) and b) are all true,transform skip-related information may be signaled and parsed beforeMTS-related information.

Condition a) transform_skip_enabled_flag==1: The decoder may checkwhether transform skip is available for (or applied to) the currentblock. The transform_skip_enabled_flag represents a flag (or syntaxelement) indicating whether transform skip is available. The flagindicating whether or not the transform skip is available may besignaled and parsed by being included in at least one of high levelsyntax such as a picture parameter set (PPS), a sequence parameter set(SPS), a video parameter set (VPS), a tile group header, and a sliceheader. If transform_skip_enabled_flag is 1, whether or not to applytransform skip may be indicated at the transform unit level.Additionally, if transform_skip_enabled_flag is 1, a maximum block sizefor which transform skip is possible may be signaled from the encoder tothe decoder. If transform_skip_enabled_flag is 0, a syntax elementindicating the transform skip may not exist (or may not be parsed) inthe transform unit syntax structure.

Condition b) tbWidth<=MaxTsSize && tbHeight<=MaxTsSize: The decoder maycheck whether the width and height of the current transform block areless than or equal to the maximum transform skip size. Iftransform_skip_enabled_flag is 1, a maximum block size for whichtransform skip is possible may be indicated, and through this, aMaxTsSize variable representing the maximum transform size may be set.As an embodiment, the transform skip may be applied to a block in whichboth the width and the height of the transform block are less than orequal to MaxTsSize.

In step S2303, if the conditions i) to v) and the conditions a) and b)described above are all true, transform_skip_flag[x0][y0], which is asyntax element indicating whether the transform skip is applied to thecurrent luma transform block, may be signaled and parsed. Iftransform_skip_flag[x0][y0] is 1, it may represent that the transform isnot applied to the current luma transform block. Iftransform_skip_flag[x0][y0] is 0, it may represent that transform isapplied to the current luma transform block. Iftransform_skip_flag[x0][y0] does not exist, transform_skip_flag[x0][y0]may be set to (or inferred as) 0.

If the conditions i) to v) described above are all true, when thefollowing conditions x), y), and z) are all true (S2304), MTS-relatedinformation may be signaled and parsed based on the transform skipinformation (S2305).

Condition x) (CuPredMode[x0][y0] !=MODE_INTRA &&sps_explicit_mts_inter_enabled_flag)∥(CuPredMode[x0][y0]==MODE_INTRA &&sps_explicit_mts_intra_enabled_flag): The decoder may check whether ornot a preset condition is satisfied on the basis of the prediction modeof the current block and whether or not explicit MTS is available. Amethod of prediction (i.e., a prediction mode) may be determined at thecoding unit level. The decoder may set the CuPredMode[x0][y0] variableon the basis of information received at the coding unit level. Theprediction mode includes intra prediction and inter prediction.Additionally, the prediction mode may include intra block copy (IBC)prediction for generating a prediction block from a reconstructed regionin the current picture. If CuPredMode[x0][y0] is MODE_INTRA, it mayrepresent that the current block is coded with intra prediction, and ifCuPredMode[x0][y0] is MODE_INTER, it may represent the current block iscoded with inter prediction. In addition, if CuPredMode[x0][y0] is MODEIBC, it may represent that the current block is coded with IBCprediction.

Meanwhile, the flag indicating whether or not MTS is available may besignaled and parsed by being included in at least one of high levelsyntax such as the picture parameter set (PPS), sequence parameter set(SPS), video parameter set (VPS), tile group header, and slice header.Specifically, whether or not MTS is available for the prediction methodmay be independently indicated. If sps_explicit_mts_inter_enabled_flagis 1, it represents that an MTS index (in this specification, it may bereferred to as tu_mts_idx, mts_idx), which is a syntax elementindicating a transform kernel for an inter predicted and/or IBCpredicted coding unit, may exist in the transform unit syntax, andwhether or not to apply MTS may be indicated depending on the MTS indexat the transform unit level. If sps_explicit_mts_inter_enabled_flag is0, it represents that the MTS index may not exist (i.e., not explicitlysignaled) in the transform unit syntax for the inter predicted and/orIBC predicted coding unit, which means that MTS may not be applied tointer predicted and/or IBC predicted blocks. In other words, ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex is not signaled to the inter predicted and/or IBC predicted codingunit, and if the MTS index does not exist, its value may be inferred as0. If sps_explicit_mts_intra_enabled_flag is 1, it represents that theMTS index may exist in the transform unit syntax for the intra-predictedcoding unit, and whether or not to apply MTS may be indicated dependingon the MTS index at the transform unit level. Ifsps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex may not exist in the transform unit syntax for the intra-predictedcoding unit, which represents that MTS may not be applied to theintra-predicted block. In other words, ifsps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex is not signaled to the intra-predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0.

In the present invention, whether or not to apply MTS may have the samemeaning as whether a transform type (or a transform kernel) other than apredetermined specific transform type (which may be referred to as abasic transform type, a default transform type, etc.) is used. If MTS isapplied, a transform type (e.g., a transform type which is any one of aplurality of transform types or in which two or more of the plurality oftransform types are combined) other than the basic transform type may beused for transform. If MTS is not applied, only the basic transform typemay be used for transform. In an embodiment, the basic transform typemay be set (or defined) as a discrete cosine transform type-2(DCT-2)-based transform type. Accordingly, the MTS index, which is asyntax element indicating a transform type applied to the currenttransform block, may include information on whether or not to apply MTS.

If the current block is coded with intra prediction, and MTS isavailable for intra prediction through high level syntax signaling(i.e., CuPredMode[x0][y0]==MODE_INTRA &&sps_explicit_mts_intra_enabled_flag) or if the current block is codedwith inter prediction or IBC prediction and MTS is available forinter/IBC prediction through high level syntax signaling(CuPredMode[x0][y0] !=MODE_INTRA &&sps_explicit_mts_inter_enabled_flag), the condition x) may be true. Thatis, in the condition x), if the corresponding MTS enabled flag is on inthe high level syntax depending on the prediction mode (that is,according to the CuPredMode[x0][y0] variable), MTS-related informationmay be signaled and parsed.

Condition y) tbWidth<=32 && tbHeight<=32: The decoder may check whetherthe width and height of the current transform block are less than orequal to 32. The maximum size of a transform block to which MTS may beapplied may be defined as 32×32. If both the width and height of thetransform block are 32 or less, MTS-related information may be signaledand parsed.

Condition z) !transform_skip_flag[x0][y0]: The decoder may check whethertransform skip is not applied to the current transform block. If thetransform skip is applied to the current transform block, transform isnot applied, and thus MTS-related information may not be signaled andparsed.

If the conditions i) to v) and the conditions x), y), and z) are alltrue, in step S2305, an MTS index, which is a syntax element indicatinga transform kernel (or transform type) applied to the horizontal andvertical directions of residual samples of the luma transform block, maybe signaled and parsed. As an embodiment, if the MTS index does notexist, it may be set to (or inferred as) 0.

According to an embodiment of the present invention, the decoder maydetermine a transform kernel (or transform type) applied to thehorizontal and vertical directions on the basis of the MTS index.Referring to FIG. 23(b), the decoder may determine a transform kernel(or transform kernel combination) applied to the horizontal and verticaldirections of residual samples of the current transform block from amongpredefined transform kernel combinations shown in FIG. 23(b) dependingon the MTS index value. In the table of FIG. 23(b), trTypeHor is avariable representing a transform kernel (or transform type) applied tothe horizontal direction of residual samples for a transform block, andtrTypeVer is a variable representing a transform kernel (or transformtype) applied to the vertical direction of residual samples for thetransform block.

If trTypeHor/trTypeVer is 0, it represents that the basic transformkernel is applied. In the present invention, the basic transform kernelmay be referred to as a first transform type. The basic transform kernelmay be DCT-II (or DCT-II-based transform kernel). If trTypeHor/trTypeVeris 1, it represents that the first transform kernel defined in thetransform kernel set is applied. In this case, the transform kernel setmay include a plurality of transform kernels in addition to the basictransform kernel (i.e., the first transform type). The first transformkernel defined in the transform kernel set may be referred to as asecond transform type. The second transform type may be DST-VII (orDST-VII-based transform kernel). If trTypeHor/trTyepeVer is 2, itrepresents that the second transform kernel defined in the transformkernel set is applied. The second transform kernel defined in thetransform kernel set may be referred to as a third transform type. Thethird transform type may be DCT-VIII (or DCT-VIII-based transformkernel).

In an embodiment of the present invention, the first bit of the MTSindex may represent whether MTS is applied to the current transformblock. If the first bit of the MTS index is 0 (that is, iftu_mts_idx[x][y] in FIG. 23(b) is 0 (that is, binarization ‘0’)), it mayrepresent that MTS is not applied to the current transform block, and inthis case, a basic kernel (or basic transform kernel) (i.e., the firsttransform type) may be applied to both the horizontal and the verticaldirections. The basic transform kernel may be DCT-II (or DCT-II-basedtransform kernel). If the first bit of the MTS index is 1 (i.e.,1<=tu_mts_idx[x][y]<=4), the second or the maximum fourth bit may beutilized to indicate the combination of the transform kernel, which isapplied to the current transform block, if MTS is applied.

For example, if the MTS index is 1 (i.e., binarization ‘10’), it mayrepresent that the second transform type is applied to both thehorizontal and the vertical directions. As described above, the secondtransform type may be DST-VII. If the MTS index is 2 (i.e., binarization‘110’), it may represent that the third transform type is applied to thehorizontal direction and the second transform type is applied to thevertical direction. As described above, the third transform type may beDCT-VIII. If the MTS index is 3 (i.e., binarization ‘1110’), it mayrepresent that the second transform type is applied to the horizontaldirection and the third transform type is applied to the verticaldirection. If the MTS index is 4 (that is, binarization ‘1111’), it mayrepresent that the third transform type is applied to both thehorizontal direction and the vertical direction.

In the embodiment described above, an example in which the MTS index issignaled by integrating the flag and the index has been described, butthe present invention is not limited thereto. As in the methodpreviously described in FIG. 15, the MTS flag is signaled first, and theMTS index may be signaled when the corresponding flag is 1. That is, ifthe conditions i) to v), the conditions x), y), and z) are all true, theMTS flag may be signaled and parsed, and if the corresponding MTS flagis 1, the MTS index may be signaled and parsed. If the MTS flag andindex do not exist, their values may be set to (or inferred as) 0,respectively. In addition, in an embodiment, the MTS index may besignaled using a fixed-length number of bits.

Meanwhile, MTS is a method of finding a combination of horizontal andvertical transform kernels optimized for the energy distribution of theresidual signal based on the transform kernel defined in the transformkernel set. If the number of transform kernels defined in the transformkernel set is N, the encoder may find an optimal transform kernelcombination (i.e., horizontal and vertical direction transform kernel)that minimizes the rate-distortion cost, after performing transform andinverse transform up to (2{circumflex over ( )}N+1) times.

Accordingly, the time complexity of the encoder may increase rapidly dueto MTS. In addition, as the size of the transform kernel increases,implementation complexity generally tends to increase. Accordingly, ifthe maximum size of a block to which MTS may be applied is signaled at ahigh level, MTS is applied only if the size of the processing block isless than or equal to the maximum size to which MTS may be applied, andthe MTS-related syntax is signaled and parsed, the encoder timecomplexity described above may be efficiently improved, overhead due tosignaling of MTS-related syntax may be reduced, and compressionperformance may be improved.

FIG. 24 is a diagram illustrating a high level syntax structure and atransform unit syntax structure according to an embodiment to which thepresent invention is applied. According to an embodiment of the presentinvention, whether or not MTS is applicable may be signaled in any oneof raw byte sequence payload (RBSP) syntaxes of a high level set such asa sequence parameter set (SPS), picture parameter set (PPS), videoparameter set (VPS), tile group header, and slice header. In FIG. 24, itis assumed a method of signaling MTS-related information through asyntax of sequence level. In describing an embodiment of the presentinvention with reference to FIG. 24, a decoder is mainly described forconvenience of description, but the MTS signaling method according tothe present embodiment may be applied substantially equally to theencoder.

Specifically, first, the decoder may parse sps_mts_enabled_flag (S2401).If sps_mts_enabled_flag is 1, it may represent that a plurality oftransform kernels including the basic transform kernel may be used. Inaddition, it may represent that sps_explicit_mts_inter_enabled_flag andsps_explicit_mts_intra_enabled_flag exist in the corresponding levelsyntax structure. The sps_explicit_mts_inter_enabled_flag andsps_explicit_mts_intra_enabled_flag represent a syntax (or syntaxelement) indicating, if inter prediction and intra prediction areapplied, whether MTS is available, respectively. If sps_mts_enabled_flagis 0, it may represent that sps_explicit_mts_intra_enabled_flag andsps_explicit_mts_inter_enabled_flag do not exist in the correspondinglevel syntax structure. That is, it may indicate that only the basictransform kernel is used for transform.

If sps_mts_enabled_flag is 1, the decoder may parsesps_explicit_mts_intra_enabled_flag andsps_explicit_mts_inter_enabled_flag (S2402). Thesps_explicit_mts_intra_enabled_flag represents a syntax elementindicating whether MTS may be applied to an intra-predicted block. Ifsps_explicit_mts_intra_enabled_flag is 1, it may represent that an MTSindex may exist in a transform unit of an intra predicted coding unit.If sps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex does not exist in the transform unit of the intra predicted codingunit, which may represent that MTS may not be applied to the intrapredicted block. The sps_explicit_mts_inter_enabled_flag is an elementindicating whether MTS may be applied to an inter predicted or IBCpredicted block. If sps_explicit_mts_inter_enabled_flag is 1, it mayrepresent that an MTS index may exist in a transform unit of an interpredicted or IBC predicted coding unit. Ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex for the inter predicted or IBC predicted coding unit does notexist, which may represent that MTS may not be applied to the interpredicted or IBC predicted block.

If sps_explicit_mts_intra_enabled_flag is 1 orsps_explicit_mts_inter_enabled_flag is 1, maximum block size informationto which MTS may be additionally applied may be signaled (S2403).Specifically, when the maximum size of a block to which MTS may beapplied is MaxMtsSize, a value, which is obtained by taking a logarithmof base 2 to MaxMtsSize and then subtracting 2, may be signaled andparsed. That is, (sps_log 2_max_explicit_mts_size_minus2+2) mayrepresent the maximum size of a block whose MTS index may be greaterthan 0 in the luma transform block. An MTS index greater than 0 mayrepresent that MTS has been applied to the corresponding transformationblock. As an embodiment, the value of sps_log2_max_explicit_mts_size_minus2 may be a value in a range of [0, 3], andif sps_log 2_max_explicit_mts_size_minus2 does not exist, it may be setto (or inferred as) 0. Depending on the received value of sps_log2_max_explicit_mts_size_minus2, the variable MaxMtsSize may be set (orinduced) 1<<(sps_log 2_max_explicit_mts_size_minus2+2).

According to an embodiment of the present invention, the decoder mayobtain MTS information at the transform unit level, based on informationobtained from the high level syntax according to the embodimentdescribed with reference to FIG. 24(a) in the syntax structure asillustrated in FIG. 24(b). For parts not described in FIG. 24(b),reference may be made to the contents previously described in FIG. 23.As described above, in FIG. 23(a), the block size the condition forsignaling and parsing the MTS index may include a case (that is,tbWidth<=32 && tbHeight<=32) where the width of the transform block is32 or less and the height of the transform block is 32 or less. That is,in this case, it may be interpreted that the maximum size of the blockto which MTS may be applied is set to a fixed value of 32 in both theencoder and the decoder.

If MTS is performed (or applied) only to a block in which a width of atransform block is 16 or less and a height of the transform block is 16or less (i.e, tbWidth<=16 && tbHeight<=16) in order to performhigh-speed encoding in the encoder, even if the size of the transformblock is greater than 16×16 (e.g., (tbWidth<=16 && tbHeight==32)(tbWidth==32 && tbHeight<=16)), the encoder should always signal the MTSindex as 0. In addition, even if the size of the transform block isgreater than 16×16, the decoder should always parse the MTS index. In anembodiment of the present invention, in order to solve inefficiency ofsuch a signaling structure, MTS may be applied based on information onthe size of the maximum transform block to which the MTS indicated at ahigh level may be applied, and a MTS-related syntax element may besignaled and parsed. As an example, from sps_log2_max_explicit_mts_size_minus2, MaxMtsSize, which is a variablerepresenting the maximum size of a transform block to which MTS may beapplied, may be determined. The encoder/decoder may apply MTS to a block(i.e., tbWidth<=MaxMtsSize && tbHeight<=MaxMtsSize) in which a width ofa transform block is MaxMtsSize or less and a height of the transformblock is MaxMtsSize or less. If this the condition is satisfied, an MTSindex may be signaled and parsed.

The encoder may perform transform and inverse transform using multiplepreset transform kernel combinations (i.e., horizontal and verticaldirection transform kernel) only if the size of the current transformblock satisfies the conditions described above, and select the optimaltransform kernel combination that minimizes rate-distortion cost. Theoptimal transform kernel combination may be signaled with an MTS indexsyntax element. If the block size the condition described above issatisfied (that is, tbWidth<=MaxMtsSize && tbHeight<=MaxMtsSize), thedecoder may determine a transform kernel applied to the horizontal andvertical directions of residual samples of the current transform blockby parsing the MTS index. If the width of the current transform block isgreater than MaxMtsSize or the height of the current transform block isgreater than MaxMtsSize (i.e., tbWidth>MaxMtsSize tbHeight>MaxMtsSize),only one transform kernel combination with which the basic kernel isapplied to both the horizontal and vertical directions may be used, andthe encoder may not signal the MTS index. The decoder may not parse theMTS index and may set its value to (or infer as) 0, and may apply abasic transform kernel to both the horizontal and the verticaldirections. As described above, the basic transform kernel may beDCT-II.

Meanwhile, IBC prediction is a method of generating a prediction blockfor a current block from a reconstructed region of the current picturein which reconstruction is being progressed and, in which, predictionmay be performed by utilizing a block at a position indicated by a blockvector of the current block in a current picture as a reference block,unlike intra prediction, in which directional or non-directionalprediction is performed using reconstructed reference samples around thecurrent block. In this case, the position indicated by the block vectormay be a reconstructed region of the current picture. In addition,unlike inter prediction in which a reference picture is a past picturein which the reference picture has already been reconstructed or afuture picture, IBC prediction may use the current picture as areference picture. As described above, IBC prediction hascharacteristics different from conventional intra prediction and interprediction, and may be considered to be similar to inter prediction inthat a prediction block is generated from a position indicated by avector, but since the reference region for prediction is a currentpicture, IBC prediction may exhibit different characteristics from interprediction.

The energy distribution characteristics (i.e., the energy distributionof the residual signal in the horizontal direction and the energydistribution of the residual signal in the vertical direction) of theresidual signal tends to change depending on the prediction method, andthus the optimal transformation kernel may vary depending on theprediction method. The DST-VII-based and DCT-VIII-based transformkernels used for inter and intra predicted blocks may not be consideredto reflect the characteristics of the residual signal of the blockpredicted by IBC well. For this reason, for a block predicted by IBC, atransform skip or DCT-II as a basic kernel may be more efficient. Inparticular, in the case of IBC prediction, it generally exhibits highefficiency during screen contents coding and is used relatively often.If screen content coding is performed, an amount of the residual signalmay be relatively small and a pattern of the residual signal may beuniform, compared to coding for a general image. In this case, if thebasic transform kernel is applied without applying MTS, not onlysignaling bits may be reduced, but transform efficiency may be improved.Accordingly, according to an embodiment of the present invention, if IBCprediction is used for a current block, MTS is not applied and transformskip or a basic transform kernel may be used. The basic transform kernelmay be DCT-II (or DCT-II-based transform kernel).

FIG. 25 is a diagram illustrating a method of selecting, if IBCprediction is applied, a transform method, as an embodiment to which thepresent invention is applied. For matters not described in theembodiment of FIG. 25, reference may be made to the content ofpreviously described in FIG. 23, and a redundant description thereofwill be omitted. Referring back to the condition x) of FIG. 23, whensps_explicit_mts_inter_enabled_flag is 1, if CuPredMode[x0][y0] isMODE_INTER or MODE IBC, that is, if inter prediction or IBC predictionis applied to the current block, a transform kernel may be determined byparsing the MTS index according to the conditions y) and z). In otherwords, not only when inter prediction is applied to the current block,but also when IBC prediction is applied to the current block, if apreset condition is satisfied, the encoder/decoder may determine thetransform kernel by signaling/parsing the MTS index.

On the other hand, referring to FIG. 25, in step S2501, ifsps_explicit_mts_inter_enabled_flag is 1, the decoder may check whetherCuPredMode[x0][y0] is MODE_INTER, and determine whether or not to parsethe MTS index. That is, when sps_explicit_mts_inter_enabled_flag is 1,if CuPredMode[x0][y0] is MODE_INTER, that is, if inter prediction isapplied to the current block, the decoder may determine the transformkernel by checking the conditions y) and z) and parsing the MTS index.According to an embodiment of the present invention, ifCuPredMode[x0][y0] is MODE IBC, the MTS index may not be signaledregardless of sps_explicit_mts_inter_enabled_flag andsps_explicit_mts_intra_enabled_flag. The decoder may not parse the MTSindex and may set its value to (or infer as) 0. Accordingly, if IBCprediction is applied to the current block, transform skip or a basictransform kernel may be applied to the horizontal and verticaldirections in the current transform unit (or transform block) on thebasis of a transform_skip_flag (i.e., transform_skip_flag[x0][y0]). Inthis case, the basic transform kernel may be DCT-II.

In addition, according to another embodiment of the present invention,if IBC prediction is applied to the current block, the encoder/decoderutilizes MTS, but a transform kernel set different from the transformkernel set used in intra and/or inter prediction may be configured.Alternatively, if IBC prediction is applied, the encoder/decoder mayconfigure a transform kernel set including all or part of the transformkernel used in intra prediction and/or inter prediction. In this case,the encoder/decoder may determine a transform kernel by signaling andparsing the MTS index with the same structure as that in FIG. 23(a)previously described.

FIG. 26 is a diagram showing an example of a transform tree syntaxstructure according to an embodiment of the present invention. Accordingto an embodiment of the present invention, a transform tree splittingmay be processed through a transform tree syntax structure. Leaf nodesof the transform tree may be transform units (TU), and parsing andprocessing of syntax elements related to the residual signal may beperformed in the transform unit. The transform tree syntax may receive(X0, y0) representing the top-left coordinate of the split node (orsplit block) in the transformation tree, tbWidth representing the widthof the split node, tbHeight representing the height of the split node,treeType representing a variable indicating the type of coding treecurrently being processed, and a chType variable which is set accordingto the treeType variable as inputs. If treeType is SINGLE_TREE, it mayrepresent that luma and chroma are coded with the same coding treestructure. When treType is DUAL_TREE_LUMA, if luma and chroma are codedwith different coding tree structures, it may represent that the treecurrently being processed is a tree for luma components. When thetreeType is DUAL_TREE_CHROMA, if the luma and chroma are coded withdifferent coding tree structures, it may represent that the treecurrently being processed is a tree for chroma components. The variablechType may be set on the basis of treeType, and if the treeType isDUAL_TREE_CHROMA, chType may be set to 1, otherwise, it may be set to 0.If the transform tree syntax (or function) is called for the first time,(x0, y0) may be the same as the top-left coordinate of the coding unit,and tbWidth, tbHeight, treeType, and chType may be the same as thewidth, height, treeType, and chType of the coding unit, respectively. Acase where the root node of the transform tree having the same size asthe coding unit is split may be classified into the following threetypes. The root node of the transform tree may correspond to the codingunit, and the leaf node of the transform tree may correspond to thetransform unit.

i) Case where the size of the root node and the split node is greaterthan the luma maximum transform size MaxTbSizeY (tbWidth>MaxTbSizeYtbHeight>MaxTbSizeY): MaxTbSizeY is a variable that represents the sizeof the luma maximum transform block and may be determined on the basisof the maximum size of the transform kernel defined in theencoder/decoder. In general, as the maximum size of the transform kernelincreases, energy compaction increases and coding efficiency may beimproved, but sizes of pipeline buffers of a hardware encoder anddecoder may be determined according to the maximum size of the transformkernel. Therefore, the maximum size of the transform kernel may bedetermined in consideration of both complexity and coding efficiency. Asan embodiment, MaxTbSizeY may be set to the same fixed constant value inboth the encoder and the decoder, or may be set according to a syntaxelement signaled at a high level. For example, maximum size informationof the luma transform block may be indicated by being included in anyone of SPS, PPS, and VPS. For example,sps_max_luma_transform_size_64_flag may be signaled by being included inthe SPS. If sps_max_luma_transform_size_64_flag is 1, it may representthat the luma maximum transform size is 64, and ifsps_max_luma_transform_size_64_flag is 0, it may represent that the lumamaximum transform size is 32. On the basis of thesps_max_luma_transform_size_64_flag, the variables MaxTbLog 2SizeY andMaxTbSizeY may be set as shown in Equation 1 below.

MaxTbLog 2SizeY=sps_max_luma_transform_size_64_flag?6:5

MaxTbSizeY=1<<MaxTbLog 2SizeY  [Equation 1]

In Equation 1, the variable MaxTbSizeY represents the maximum size ofthe luma transform block, and MaxTbLog 2SizeY is a variable representinga value obtained by taking the logarithm of base 2 to MaxTbSizeY. In theembodiment described above, the maximum size information (or syntaxelement) of the luma transform block may be signaled in the form of aflag at the SPS level. In describing the embodiment of the presentinvention, it has been described that the maximum size of the lumatransform block is determined to be either 64 or 32, but the presentinvention is not limited thereto. For example, a value obtained bytaking the logarithm of base 2 to the maximum size of the luma transformblock may be directly signaled. As an embodiment, MaxTbLo2SizeY andMaxTbSizeY may be determined at a sequence level, and as anotherexample, the parameter (or variable) may be determined at a picture,tile, or slice level lower than the sequence level.

In one embodiment, in the transform tree splitting based on the maximumsize of luma transform block, a splitting method (or split-relatedinformation) may not be explicitly signaled. On the other hand, in thecase of ISP or SBT, the splitting method for the transform tree may beexplicitly signaled, and splitting of the transform tree based on theluma maximum transform size may be performed if ISP and SBT are notapplied. Accordingly, referring to FIG. 26, in step S2601, the decodermay check whether ISP or SBT is applied to the current coding unit. IfIntraSubPartitionsSplitType is ISP_NO_SPLIT, it may represent that ISPis not applied to the current coding unit. IfIntraSubPartitionsSplitType is not ISP_NO_SPLIT, it may represent thatISP is applied to the current coding unit. In addition, if cu_sbt_flagis 1, it may represent that SBT is applied to the current coding unit,and if cu_sbt_flag is 0, it may represent that SBT is not applied to thecurrent coding unit. As described above, the transform tree splittingbased on the luma maximum transform size may be performed if ISP and SBTare not applied, and thus if IntraSubPartitionsSplitType is ISP_NO_SPLITand cu_sbt_flag is 0 in the if statement of step S2601, the transformtree splitting may be performed on the basis of the luma maximumtransform size.

In one embodiment of the present invention, if the width tbWidth orheight tbHeight of the root node and the split intermediate node of thetransform tree is greater than the luma maximum transform size, theencoder/decoder does not explicitly signal/parse information on thesplitting method, and may split the transform tree until both the widthand height of the split node become less than or equal to the lumamaximum transform size. If both the width tbWidth and the heighttbHeight of the current node are less than or equal to the luma maximumtransform size, the current node may be determined as a leaf node of thetransform tree, that is, a transform unit.

For example, if the luma maximum transform size is 32 for a 64×64 rootnode (same size as the coding unit), the root node of the transform treeis split into four nodes each having a size 32×32, and a leaf nodehaving a size of 32×32 may be determined as a transform unit withoutadditional splitting. As another example, if the luma maximum transformsize for a 64×32 root node is 32, the root node of the transform tree issplit into two nodes each having a size 32×32, and a leaf node having asize of 32×32 may be determined as a transform unit without additionalsplitting. If the width or height of the root node and/or the splitintermediate node of the transform tree is greater than the root node,the encoder/decoder may split the transform tree by recursively callingthe transform tree syntax. If the width and height of the root nodeand/or the split intermediate node of the transform tree are all lessthan or equal to the luma maximum transform size, the corresponding nodemay be determined as a transform unit without additional splitting, andparsing and processing of a syntax element related to a residual signalmay be performed in the transform unit syntax structure.

ii) Case SBT is applied to the current coding unit (cu_sbt_flag==1):Unlike the case i) described above, if SBT is applied, the transformtree splitting method may be determined through explicit signaling of asyntax element at the coding unit level.

iii) If ISP is applied to the current coding unit(IntraSubPartitionsSplitType !=ISP_NO_SPLIT): Unlike the case i)described above, if ISP is applied, the transform tree splitting methodmay be determined through explicit signaling of the syntax element atthe coding unit level. For example, if IntraSubPartitionsSplitType isISP HOR SPLIT, it may mean horizontal splitting by ISP, and ifIntraSubPartitionsSplitType is ISP VER SPLIT, it may mean verticalsplitting by ISP.

As in the cases of i), ii), and iii) described above, if the transformtree may be classified into the case i) where it is split withoutexplicit signaling of a syntax element, and the cases ii) and iii) wherethe splitting method is indicated using explicit signaling of a syntaxelement. According to the transform tree syntax structure shown in FIG.26, all of the transform tree splitting described in cases i), ii), andiii) may be performed (or applied). The transform tree syntax structureof FIG. 26 may be summarized as shown in Table 15 below.

TABLE 15 transform_tree( ) {if(IntraSubPartitionsSplitType==ISP_NO_SPLIT && !cu_sbt_flag) { implicitsplitting on the basis of block size } else if(cu_sbt_flag) { explicitsplitting on the basis of SBT mode } elseif(IntraSubPartitionsSplitType==ISP_HOR_SPLIT) { explicit splitting onthe basis of ISP mode } elseif(IntraSubPartitionsSplitType==ISP_VER_SPLIT) { explicit splitting onthe basis of ISP mode } }

Referring to Table 15, first, if IntraSubPartitionsSplitType isISP_NO_SPLIT and cu_sbt_flag is 0, implicit splitting may be performedon the basis of size information of the current node and the lumamaximum transform block size. Next, if IntraSubPartitionsSplitType isISP_NO_SPLIT and cu_sbt_flag is 1, the encoder/decoder may split thetransform tree based on the syntax element related to the SBTsignaled/parsed at the coding unit level. Next, ifIntraSubPartitionsSplitType is not ISP_NO_SPLIT and cu_sbt_flag is 0,horizontal splitting or vertical splitting of ISP may be performed basedon IntraSubPartitionsSplitType.

FIG. 27 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention. According to anembodiment of the present invention, the MTS index, which is a syntaxelement indicating transform skip-related information and/or a transformkernel, may be processed in the transform unit syntax structure.Referring to FIG. 27, an embodiment of the present invention isdescribed mainly on a decoder for convenience of description, but thepresent invention is not limited thereto, and transform skip and MTSdetermination method according to the present embodiment may be appliedsubstantially equally to the encoder.

Referring to FIG. 27, the transform unit syntax structure represents asyntax structure related to a transform unit for which a transformprocess is performed. The transform unit syntax may receive x0, y0,tbWidth, tbHeight, treeType, subTuIndex, and chType as inputs. (x0, y0)may represent the position of the top-left luma sample of the currenttransform block, and may be a relative position with the position of thetop-left luma sample of the current picture as a reference. Thevariables tbWidth and tbHeight represent the width and height of thecurrent transform block, respectively. A coding tree structure for theluma component and a coding tree structure for the chroma component maybe coded with the same structure or different structures. Accordingly,the treeType variable may indicate the type of coding tree currentlybeing processed. If the treeType is SINGLE_TREE, it means that the lumacomponent and the chroma component are coded with the same coding treestructure. In this case, the transform unit may include a luma transformblock and a chroma transform block according to a color format. When thetreeType is DUAL_TREE_LUMA, if the luma component and the chromacomponent are coded with a coding tree structure having differentstructures, it may represent that the coding tree currently beingprocessed is a coding tree of the luma component. In this case, thetransform unit may include the luma transform block. When the treeTypeis DUAL_TREE_CHROMA, if the luma component and the chroma component arecoded with a coding tree structure having different structures, it mayrepresent that the coding tree currently being processed is a codingtree of the chroma component. In this case, the transform unit mayinclude the chroma transform block according to the color format. ThesubTuIndex represents an index of transform unit currently beingprocessed. The chType may be set on the basis of the treeType. If thetreeType is DUAL_TREE_CHROMA, the chType may be set to 1, otherwise, itmay be set to 0.

In the transform unit syntax structure (i.e., transform unit( ), first,the encoder may indicate whether the current transform block has one ormore non-zero transform coefficients by signaling a coded block flag CBFfor a luma block, a Cb chroma block, and a Cr chroma block. Similarly,the decoder may determine (or induce) whether the current transformblock has one or more non-zero transform coefficients by parsing thecoded block flag CBF for the luma block, Cb block (or Cb chroma block),and Cr block (or Cr chroma block). For example, if tu_cbf_luma[x0][y0]is 1, it may represent that the current luma transform block includesone or more non-zero transform coefficients, and if tu_cbf_luma[x0][y0]is 0, it may represent that all transform coefficients of the currentluma transform block are 0. If tu_cbf_cb[x0][y0] is 1, it may representthat the current Cb transform block includes one or more non-zerotransform coefficients, and if tu_cbf_cb[x0][y0] is 0, it may representthat all transform coefficients of the current Cb transform block are 0.If tu_cbf_cr[x0][y0] is 1, it may represent that the current Crtransform block includes one or more non-zero transform coefficients,and if tu_cbf_cr[x0][y0] is 0, it may represent that all transformcoefficients of the current Cr transform block are 0.

In step S2701 (i.e., the first the conditional statement), the decodermay check (or examine) a preset condition for determining whether or notto apply transform skip and/or MTS. As an example, if the following theconditions i), ii), iii), iv), and v) are all true, the decoder mayrespectively parse (S2703, S2705) transform skip and MTS information(i.e., syntax elements) according to the second the conditionalstatement of step S2702 and the third the conditional statement of stepS2704, which will be described later.

Condition i) tu_cbf_luma[x0][y0]==1: The decoder may check whether thevalue of tu_cbf_luma[x0][y0] is 1. If the value of tu_cbf_luma[x0][y0]is 0, it represents that all transform coefficients of the current lumatransform block are 0, and thus the decoder may reconstruct the residualsignal to zero without performing inverse quantization and inversetransform. Accordingly, if the value of tu_cbf_luma[x0][y0] is 0,transform skip and MTS information may not be signaled and parsed.

Condition ii) treeType !=DUAL_TREE_CHROMA: The decoder may check whetherthe tree type is a tree type including a luma transform block. Bothtransform skip and MTS may be applied only to the luma transform block,and thus if the current transform unit (or transform block) does notinclude the luma transform block, transform skip and MTS information maynot be signaled and parsed. That is, if the tree type isDUAL_TREE_CHROMA, the current transform unit does not include the lumatransform block, and thus transform skip and MTS information may not besignaled and parsed.

Condition iii) (tbWidth<=32) && (tbHeight<=32): The decoder may checkwhether the width and height of the current transform block are lessthan or equal to 32. The maximum size of the transform block to whichtransform skip and MTS may be applied is 32×32, and thus if the width ofthe transform block and both are 32 or less, signaling and parsing oftransform skip and MTS information may be performed depending onadditional the conditions.

Condition iv) IntraSubPartitionsSplitType[x0][y0]==ISP_NO_SPLIT: Thedecoder may check whether ISP is not applied to the current coding unit(or coding block). If ISP is applied to the current coding unit,transform and inverse transform may be performed according to a presetmethod. Accordingly, if ISP is applied to the current coding unit,transform skip and MTS information may not be signaled and parsed. TheIntraSubPartitionsSplitType represents the variable indicating the blocksplitting method by ISP. The IntraSubPartitionsSplitType may indicatethe block splitting method by ISP according to ISP information signaledat the coding unit level. If IntraSubPartitionsSplitType is notISP_NO_SPLIT, it may represent that ISP is applied to the current block.Accordingly, if IntraSubPartitionsSplitType is not ISP_NO_SPLIT,transform skip and MTS information may not be signaled and parsed.

Condition v) !cu_sbt_flag: The decoder may check whether sub-blocktransform (SBT) is not applied to the current coding unit. If SBT isapplied to the current coding unit, transform and inverse transform maybe performed according to a preset method. Accordingly, if SBT isapplied to the current coding unit, transform skip and MTS informationmay not be signaled and parsed. The cu_sbt_flag represents a syntaxelement indicating whether SBT is applied to the current coding unit.The cu_sbt_flag may be signaled and parsed at the coding unit level. Ifcu_sbt_flag is 1, it may represent that SBT is applied to the currentcoding unit, and transform skip and MTS information may not be signaledand parsed.

If the conditions i) to v) described above are all true, in step S2702,the decoder may check (or examine) the following the conditions a), b),and c) for determining whether or not to apply the transform skip. As anexample, if the following the conditions a), b), and c) are all true,transform skip-related information may be signaled and parsed.

Condition a) sps transform_skip_enabled_flag==1: The decoder may checkwhether transform skip may be available for (or applied to) the currentblock. The sps transform_skip_enabled_flag represents a flag (or syntaxelement) indicating whether transform skip is available. The flagindicating whether or not the transform skip is available may besignaled and parsed by being included in at least one of high levelsyntax such as a picture parameter set (PPS), a sequence parameter set(SPS), a video parameter set (VPS), a tile group header, and a sliceheader. If sps transform_skip_enabled_flag is 1, whether or not to applytransform skip may be indicated at the transform unit level.Additionally, if sps transform_skip_enabled_flag is 1, the maximum blocksize for which transform skip is possible may be signaled from theencoder to the decoder. If sps transform_skip_enabled_flag is 0, asyntax element indicating the transform skip may not exist (or may notbe parsed or may not be explicitly signaled) in the transform unitsyntax structure.

Condition b) !BdpcmFlag[x0][y0]: As one of the intra prediction methods,a block-based delta pulse code modulation (BDPCM) method may be defined.If the BDPCM mode is applied, the encoder/decoder may perform intraprediction using a preset prediction mode, apply transform skip, andthen express a quantized coefficient as a sum of a sample at a previousposition and a sample at a current position according to a preset mode.That is, in the BDPCM mode, transform is not applied, and a differencebetween the sample at the current position and the sample at theprevious position, as the quantized coefficient, may be transmitted fromthe encoder to the decoder according to the preset mode. BDPCM-relatedinformation (or syntax element) may be indicated at the coding unitlevel. As an embodiment, BdpcmFlag[x0][y0] may be transmitted from theencoder to the decoder. If BdpcmFlag[x0][y0] is 1, it may represent thatBDPCM is applied to the current block. If BDPCM is currently applied, itmay be seen that transform skip is applied, and thustransform_skip_flag[x0][y0], which is a syntax element representingwhether transform skip is applied to the current luma transform block,is not explicitly signaled and may be set to (or inferred as) 1.

Condition c) tbWidth<=MaxTsSize && tbHeight<=MaxTsSize: The decoder maycheck whether the width and height of the transform block are less thanor equal to the maximum transform skip size. If spstransform_skip_enabled_flag is 1, the maximum block size for whichtransform skip is possible may be indicated, and through this, aMaxTsSize variable representing the maximum transform size may be set.As an embodiment, the transform skip may be applied to a block in whichboth the width and the height of the transform block are less than orequal to MaxTsSize.

In step S2703, if the conditions i) to v) and the conditions a), b), andc) described above are all true, transform_skip_flag[x0][y0], which is asyntax element indicating whether the transform skip is applied to thecurrent luma transform block, may be signaled and parsed. Iftransform_skip_flag[x0][y0] is 1, it may represent that the transform isnot applied to the current luma transform block. Iftransform_skip_flag[x0][y0] is 0, it may represent that transform isapplied to the current luma transform block. If transform_skip_flag[x0][y0] does not exist (i.e., is not explicitly signaled), a value oftransform_skip_flag[x0][y0] may be set (or inferred) according toBdpcmFlag[x0][y0]. If BdpcmFlag[x0][y0] is 1, thetransform_skip_flag[x0][y0] may be set to (or inferred as) 1, and ifBdpcmFlag[x0][y0] is 0, transform_skip_flag[x0][y0] may be set to (orinferred as) 0.

When the conditions i) to v) described above are all true, if thefollowing the conditions x and y are all true (S2704), MTS-relatedinformation may be signaled and parsed (S2705).

Condition x) (CuPredMode[chType][x0][y0] !=MODE_INTER &&sps_explicit_mts_inter_enabled_flag)∥(CuPred[chType]Mode[x0][y0]MODE_INTRA&& sps_explicit_mts_intra_enabled_flag): The decoder may check whether apreset condition is satisfied on the basis of the prediction mode of thecurrent block and whether or not explicit MTS is available. A method ofprediction (i.e., a prediction mode) may be determined at the codingunit level, and a variable CuPredMode[chType][x0][y0] may be set on thebasis of information received at the coding unit level. The predictionmode includes intra prediction and inter prediction. Additionally, theprediction mode may include intra block copy (IBC) prediction forgenerating a prediction block from a reconstructed region in the currentpicture. If CuPredMode[chType][x0][y0] is MODE_INTRA, it may representthat the current block is coded with intra prediction, and ifCuPred[chType]Mode[x0][y0] is MODE_INTER, it may represent the currentblock is coded with inter prediction. In addition, ifCuPredMode[chType][x0][y0] is MODE IBC, it may represent that thecurrent block is coded with IBC prediction.

Meanwhile, the flag indicating whether or not MTS is available may besignaled and parsed by being included in at least one of high levelsyntax such as the picture parameter set (PPS), sequence parameter set(SPS), video parameter set (VPS), tile group header, and slice header.Specifically, whether or not MTS is available for the prediction methodmay be independently indicated. If sps_explicit_mts_inter_enabled_flagis 1, it represents that an MTS index (in this specification, it may bereferred to as tu_mts_idx, mts_idx), which is a syntax elementindicating a transform kernel for an inter-predicted coding unit, mayexist in the transform unit syntax, and whether or not to apply MTS maybe indicated depending on the MTS index at the transform unit level. Ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex may not exist (i.e., not explicitly signaled) in the transformunit syntax for the inter predicted, which means that MTS may not beapplied to inter predicted block. In other words, ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex is not signaled to the inter predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. If MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel. If sps_explicit_mts_intra_enabled_flag is1, it represents that the MTS index may exist in the transform unitsyntax for the intra-predicted coding unit, and whether or not to applyMTS may be indicated depending on the MTS index at the transform unitlevel. If sps_explicit_mts_intra_enabled_flag is 0, it represents thatthe MTS index may not exist in the transform unit syntax for theintra-predicted coding unit. In other words, ifsps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex is not signaled to the intra-predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. When MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel.

If the current block is coded with intra prediction and MTS is availablefor intra prediction through high level syntax signaling (i.e.,CuPredMode[x0][y0] MODE_INTRA && sps_explicit_mts_intra_enabled_flag),or if the current block is coded with inter prediction and MTS isavailable for inter prediction through high level syntax signaling(i.e., CuPredMode[chTye][x0][y0]==MODE_INTER &&sps_explicit_mts_inter_enabled_flag), the condition x) may be true. Thatis, in the condition x), if the corresponding MTS enabled flag is on inthe high level syntax according to the prediction mode (that is,according to the CuPred[chType]Mode[x0][y0] variable), MTS-relatedinformation may be signaled and parsed.

Condition y) !transform_skip_flag[x0][y0]: The decoder may check whethertransform skip is not applied to the current transform block. If thetransform skip is applied to the current transform block, transform isnot applied, and thus MTS-related information may not be signaled andparsed.

If the conditions i) to v) and the conditions x) and y) are all true, instep S2705, an MTS index, which is a syntax element indicating atransform kernel (or transform type) applied to the horizontal andvertical directions of residual samples of the luma transform block, maybe signaled and parsed. As an embodiment, if the MTS index does notexist, the MTS index may be set to (or inferred as) 0. This mayrepresent that the preset basic kernel is applied to both the horizontaldirection and the vertical direction, and the basic kernel may be aDCT-II-based integer kernel. The transform kernel applied to thehorizontal direction and the vertical direction according to the MTSindex may follow the method previously described in FIG. 14.

On the other hand, in the transform unit syntax structure of FIG. 27,even if the transform tree is split into multiple transform units on thebasis of the maximum size (MaxTbSizeY) of the luma transform block, theMTS index may be parsed in each of the split transform units. Inaddition, a different transform kernel may be applied to each transformunit on the basis of the parsed MTS index. For example, if the lumamaximum transform size is 32, a coding unit having a size of 64×64 maybe split into four transform units each having a size of 32×32. Each ofthe split transform units always has a tbWidth of 32 or less(tbWidth<=32) and a tbHeight of 32 or less (tbHeight<=32), and thus theMTS index may be signaled and parsed if the remaining the conditions forsignaling/parsing the MTS index are true. In this case, each splittransform unit may have a different MTS index value. That is, this meansthat the transform kernels applied to the horizontal direction and thevertical direction of each split transform unit may be different. Forexample, it may be possible to apply a DCT-II-based kernel to thehorizontal direction and the vertical direction of the split firsttransform block and apply a DST-VII-based kernel may be applied to thehorizontal direction and vertical direction of the split secondtransform block.

As described above, if different transform kernels are applied to eachof the split transform units according to the MTS index, there is anadvantage in that an optimal transform kernel may be selected accordingto characteristics of a residual signal included in the transform unit.However, the prediction method is determined at the coding unit level,and thus the characteristics of the residual signals included in thetransform units split at the coding unit level are highly likely to besimilar to each other. Accordingly, if the same transform kernel isapplied to all of the divided transform units by setting the MTS indexto a preset value without explicitly signaling the MTS to the transformunits split based on the maximum size of the luma transform block,signaling overhead required for indicating the transform kernel may bereduced. That is, even if the (tbWidth<=32) && (tbHeight<=32) thecondition is true in the transform unit split by the maximum size of theluma transform block, the MTS index may not be explicitly signaled, andthe MTS index may be set to a preset value. The preset value may be 0,and in this case, a DCT-II-based basic kernel may be applied to both thehorizontal direction and the vertical direction.

FIG. 28 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention. For matters notdescribed in the embodiment of FIG. 28, reference may be made to thecontent of FIG. 27 previously described, and redundant descriptionsthereof will be omitted. According to an embodiment of the presentinvention, the MTS index, which is a syntax element indicating transformskip-related information and/or a transform kernel, may be processed inthe transform unit syntax structure. Referring to FIG. 28, an embodimentof the present invention is described mainly on a decoder forconvenience of description, but the present invention is not limitedthereto, and transform skip and MTS determination method according tothe present embodiment may be applied substantially equally to theencoder.

In step S2801 (i.e., the first the conditional statement), the decodermay check (or examine) a preset condition for determining whether or notto apply transform skip and/or MTS. As an example, if the following theconditions i), ii), iii), and iv) are all true, the decoder mayrespectively parse (S2703, S2705) transform skip and MTS information(i.e., syntax element) according to the second the conditional statementof step S2802 and the third the conditional statement of step S2804,which will be described later.

Condition i) tu_cbf_luma[x0][y0]==1: The decoder may check whether thevalue of tu_cbf_luma[x0][y0] is 1. If the value of tu_cbf_luma[x0][y0]is 0, it represents that all transform coefficients of the current lumatransform block are 0, and thus the decoder may reconstruct the residualsignal to zero without performing inverse quantization and inversetransform. Accordingly, if the value of tu_cbf_luma[x0][y0] is 0,transform skip and MTS information may not be signaled and parsed.

Condition ii) treeType !=DUAL_TREE_CHROMA: The decoder may check whetherthe tree type is a tree type including a luma transform block. Bothtransform skip and MTS may be applied only to the luma transform block,and thus if the current transform unit (or transform block) does notinclude the luma transform block, transform skip and MTS information maynot be signaled and parsed. That is, if the tree type isDUAL_TREE_CHROMA, the current transform unit does not include the lumatransform block, and thus transform skip and MTS information may not besignaled and parsed.

Condition iii) IntraSubPartitionsSplitType[x0][y0]==ISP_NO_SPLIT: Thedecoder may check whether ISP is not applied to the current coding unit(or coding block). If ISP is applied to the current coding unit,transform and inverse transform may be performed according to a presetmethod. Accordingly, if ISP is applied to the current coding unit,transform skip and MTS information may not be signaled and parsed. TheIntraSubPartitionsSplitType represents the variable indicating the blocksplitting method by ISP. The IntraSubPartitionsSplitType may indicatethe block splitting method by ISP according to ISP information signaledat the coding unit level. If IntraSubPartitionsSplitType is notISP_NO_SPLIT, it may represent that ISP is applied to the current block.Accordingly, if IntraSubPartitionsSplitType is not ISP_NO_SPLIT,transform skip and MTS information may not be signaled and parsed.

Condition iv) ! cu_sbt_flag: The decoder may check whether sub-blocktransform (SBT) is not applied to the current coding unit. If SBT isapplied to the current coding unit, transform and inverse transform maybe performed according to a preset method. Accordingly, if SBT isapplied to the current coding unit, transform skip and MTS informationmay not be signaled and parsed. The cu_sbt_flag represents a syntaxelement indicating whether SBT is applied to the current coding unit.The cu_sbt_flag may be signaled and parsed at the coding unit level. Ifcu_sbt_flag is 1, it may represent that SBT is applied to the currentcoding unit, and transform skip and MTS information may not be signaledand parsed.

If the conditions i) to iv) described above are all true, in step S2802,the decoder may check (or examine) the following the conditions a), b),and c) for determining whether or not to apply the transform skip. As anexample, if the following the conditions a), b), and c) are all true,transform skip-related information may be signaled and parsed. Theconditions a), b), and c) described above may be defined to be the sameas the conditions a), b), and c) previously described in FIG. 27.

In step S2803, if the conditions i) to iv) and the conditions a), b),and c) described above are all true, transform_skip_flag[x0][y0], whichis a syntax element indicating whether the transform skip is applied tothe current luma transform block, may be signaled and parsed. Iftransform_skip_flag[x0][y0] is 1, it may represent that transform is notapplied to the current luma transform block. Iftransform_skip_flag[x0][y0] is 0, it may represent that transform isapplied to the current luma transform block. Iftransform_skip_flag[x0][y0] does not exist, a value oftransform_skip_flag[x0][y0] may be set (or inferred) according toBdpcmFlag[x0][y0]. If BdpcmFlag[x0][y0] is 1,transform_skip_flag[x0][y0] may be set to (or inferred as) 1, and ifBdpcmFlag[x0][y0] is 0, transform_skip_flag[x0][y0] may be set to (orinferred as) 0.

According to an embodiment of the present invention, even if thetransform tree is split into multiple transform units according to themaximum size the condition of the luma transform block (i.e., implicitlysplit on the basis of the luma transform maximum size), if all of theconditions i) to iv) and the conditions a), b), and c) are satisfied,the decoder may parse transform_skip_flag[x0][y0] and apply transformskip to the split transform units according to the value oftransform_skip_flag[x0][y0].

When the conditions i) to iv) described above are all true, if thefollowing the conditions x), y), and z) are all true (S2804),MTS-related information may be signaled and parsed (S2805).

Condition x) (CuPredMode[chType][x0][y0] !=MODE_INTER &&sps_explicit_mts_inter_enabled_flag)∥(CuPredMode[chType][x0][y0]==MODE_INTRA&& sps_explicit_mts_intra_enabled_flag): The decoder may check whetheror not a preset condition is satisfied on the basis of the predictionmode of the current block and whether or not explicit MTS is available.A method of prediction (i.e., a prediction mode) may be determined atthe coding unit level, and a variable CuPredMode[chType][x0][y0] may beset on the basis of information received at the coding unit level. Theprediction mode includes intra prediction and inter prediction.Additionally, the prediction mode may include intra block copy (IBC)prediction for generating a prediction block from a reconstructed regionin the current picture. If CuPred[chType]Mode[x0][y0] is MODE_INTRA, itmay represent that the current block is coded with intra prediction, andif CuPred[chType]Mode[x0][y0] is MODE_INTER, it may represent thecurrent block is coded with inter prediction. In addition, ifCuPred[chType]Mode[x0][y0] is MODE IBC, it may represent that thecurrent block is coded with IBC prediction.

Meanwhile, the flag indicating whether or not MTS is available may besignaled and parsed by being included in at least one of high levelsyntax such as the picture parameter set (PPS), sequence parameter set(SPS), video parameter set (VPS), tile group header, and slice header.Specifically, whether or not MTS is available for the prediction methodmay be independently indicated. If sps_explicit_mts_inter_enabled_flagis 1, it represents that an MTS index (in this specification, it may bereferred to as tu_mts_idx, mts_idx), which is a syntax elementindicating a transform kernel for an inter predicted coding unit, mayexist in the transform unit syntax, and whether or not to apply MTS maybe indicated depending on the MTS index at the transform unit level. Ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex may not exist (i.e., not explicitly signaled) in the transformunit syntax for the inter predicted coding unit, which means that MTSmay not be applied to inter predicted blocks. In other words, ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex is not signaled to the inter predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. If MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel. If sps_explicit_mts_intra_enabled_flag is1, it represents that the MTS index may exist in the transform unitsyntax for the intra-predicted coding unit, and whether or not to applyMTS may be indicated depending on the MTS index at the transform unitlevel. If sps_explicit_mts_intra_enabled_flag is 0, it represents thatthe MTS index may not exist in the transform unit syntax (not explicitlysignaled) for the intra-predicted coding unit. In other words, ifsps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex is not signaled to the intra-predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. If MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel.

If the current block is coded with intra prediction, and MTS isavailable for intra prediction through high level syntax signaling(i.e., CuPred[chType]Mode[x0][y0] MODE_INTRA &&sps_explicit_mts_intra_enabled_flag) or if the current block is codedwith inter prediction and MTS is available for inter prediction throughhigh level syntax signaling (CuPredMode[chType][x0][y0] !=MODE_INTRA &&sps_explicit_mts_inter_enabled_flag), the condition x) may be true. Thatis, in the condition x), if the corresponding MTS enabled flag is on inthe high level syntax according to the prediction mode (that is,according to the CuPred[chType]Mode[x0][y0] variable), MTS-relatedinformation may be signaled and parsed.

Condition y) !transform_skip_flag[x0][y0]: The decoder may check whethertransform skip is not applied to the current transform block. If thetransform skip is applied to the current transform block, transform isnot applied, and thus MTS-related information may not be signaled andparsed.

Condition z) (CbWidth[chType][x0][y0]<=32) &&(CbHeight[chType][x0][y0]<=32): CbWidth[chType][x0][y0] andCbHeight[chType][x0][y0] may represent the width and height of thecurrent coding block, respectively. The transform unit may be split bythe transform tree using the coding unit as the root node, and thus itmay satisfy the relationship of CbWidth[chType][x0][y0]>=tbWidth,CbHeight[chType][x0][y0]>=tbHeight. That is, according to an embodimentof the present invention, in the transform unit syntax structure of FIG.28, the MTS index may be explicitly signaled only if both the width andheight of the coding block, which is the root node, in the transformtree structure are less than or equal to a preset value. In addition, asan embodiment, the preset value may be 32. Accordingly, theencoder/decoder does not signal/parse the MTS index for the transformunit split according to the maximum size the condition of the lumatransform block, and may set the MTS index to (or infer as) a presetvalue of 0. In this case, a DCT-II-based basic transform kernel may beapplied to the horizontal direction and vertical direction.

In one embodiment, assuming that the luma maximum transform size is 64and both ISP and SBT are not applied to the current block, when both thewidth and height of the coding block are 32 or less, the transform treeis not split, and the size of the transform unit may be determined to bethe same as the size of the coding unit. In this case, both the widthand height of the transform block are guaranteed to be 32 or less, andthus the decoder may parse the MTS index and determine/use the transformkernel according to the MTS index value.

In addition, in an embodiment, assuming that the luma maximum transformsize is 32 and both ISP and SBT are not applied to the current block, ifboth the width and height of the coding block are 32 or less, thetransform tree is not split, and the size of the transform unit may bedetermined to be the same as the size of the coding unit. Accordingly,both the width and height of the transform block are guaranteed to be 32or less, and thus the decoder may parse the MTS index and determine/usethe transform kernel according to the MTS index value. On the otherhand, even if both the width and height of the transform block are 32 orless, if the width or height of the coding block is greater than 32,this indicates that it is a transform unit split according to thecondition of the maximum size of the luma transform block, and thus theMTS index is not parsed and may be set to (or inferred as) a presetvalue. That is, even if the width and height of the transform block areboth less than 32, if the width or height of the coding block is greaterthan 32, the MTS index is not parsed and may be set to (or inferred as)a preset value. The preset value may be 0, and in this case, aDCT-II-based basic kernel may be applied to both the horizontaldirection and the vertical direction.

If the conditions i) to iv) and the conditions x), y), and z) describedabove are all true, in step S2805, an MTS index, which is a syntaxelement indicating a transform kernel (or transform type) applied to thehorizontal and vertical directions of residual samples of the lumatransform block, may be signaled and parsed. As an embodiment, if theMTS index does not exist, the MTS index may be set to (or inferred as)0. This may represent that the preset basic kernel is applied to boththe horizontal direction and the vertical direction, and the basickernel may be a DCT-II-based integer kernel. The transform kernelapplied to the horizontal direction and the vertical direction accordingto the MTS index may follow the method previously described in FIG. 14.

FIG. 29 is a diagram illustrating a transform unit syntax structureaccording to an embodiment of the present invention. For matters notdescribed in the embodiment of FIG. 29, reference may be made to thecontent of FIG. 27 described above, and redundant descriptions thereofwill be omitted. In the transform unit syntax structure of FIG. 29, atrafoDepth variable may be additionally received as an input. Here,trafoDepth is a variable representing the degree of splitting (orsplitting depth) of a transform tree. For example, if trafoDepth is 0,it may represent that the transform tree is not split, and in this case,the size of the coding unit and the transform unit may be the same. IftrafoDepth is greater than 0, it may represent that the transform treeis split by using the coding unit as a root node, and in this case, thecoding unit may include a plurality of transform units. For thecomputation of trafoDepth, the transform tree syntax structure may alsoreceive the trafoDepth variable as input, and if the transform treesyntax structure is executed for the first time, trafoDepth as 0 may bereceived as an input.

In the transform tree syntax structure previously described in FIG. 26,when IntraSubPartitionsSplitType is ISP_NO_SPLIT and cu_sbt_flag[x0][y0]is 0, if the transform tree is split according to the maximum size thecondition of the luma transform block (i.e., tbWidth>MaxTbSizeYtbHeight>MaxTbSizeY), the encoder/decoder may trafoDepth to trafoDepth+1(i.e., (current splitting depth+1)) in performing a transform treeprocess (or syntax). On the other hand, when IntraSubPartitionsSplitTypeis ISP_NO_SPLIT and cu_sbt_flag[x0][y0] is 0, if the transform tree isnot split (i.e., tbWidth<=MaxTbSizeY && tbHeight<=MaxTbSizeY), thetransform unit syntax structure may be performed, and theencoder/decoder may set trafoDepth of the transform unit to trafoDepth(i.e., set equal to the current split depth). If the transform tree issplit by applying SBT or ISP, in performing a transform unit processingprocess, the encoder/decoder may set trafoDepth to trafoDepth+1((current split depth+1)).

In step S2901 (i.e., the first the conditional statement), the decodermay check (or examine) a preset condition for determining whether or notto apply transform skip and/or MTS. As an example, if the following theconditions i), ii), iii), iv), and v) are all true, the decoder mayrespectively parse (S2903, S2905) transform skip and MTS information(i.e., syntax element) according to the second the conditional statementof step S2902 and the third the conditional statement of step S2904,which will be described later.

Condition i) tu_cbf_luma[x0][y0] 1: The decoder may check whether avalue of tu_cbf_luma[x0][y0] is 1. If the value of tu_cbf_luma[x0][y0]is 0, it represents that all transform coefficients of the current lumatransform block are 0, and thus the decoder may reconstruct the residualsignal to zero without performing inverse quantization and inversetransform. Accordingly, if the value of tu_cbf_luma[x0][y0] is 0,transform skip and MTS information may not be signaled and parsed.

Condition ii) treeType !=DUAL_TREE_CHROMA: The decoder may check whetherthe tree type is a tree type including a luma transform block. Bothtransform skip and MTS may be applied only to the luma transform block,and thus if the current transform unit (or transform block) does notinclude the luma transform block, transform skip and MTS information maynot be signaled and parsed. That is, if the tree type isDUAL_TREE_CHROMA, the current transform unit does not include the lumatransform block, and thus transform skip and MTS information may not besignaled and parsed.

Condition iii) (tbWidth<=32) && (tbHeight<=32): The decoder may checkwhether the width and height of the current transform block are lessthan or equal to 32. The maximum size of the transform block to whichtransform skip and MTS may be applied is 32×32, and thus if both thewidth and height of the transform block are 32 or less, signaling andparsing of transform skip and MTS information may be performed accordingto additional the conditions.

Condition iv) IntraSubPartitionsSplitType[x0][y0]==ISP_NO_SPLIT: Thedecoder may check whether ISP is not applied to the current coding unit(or coding block). If ISP is applied to the current coding unit,transform and inverse transform may be performed according to a presetmethod. Accordingly, if ISP is applied to the current coding unit,transform skip and MTS information may not be signaled and parsed. TheIntraSubPartitionsSplitType represents the variable indicating the blocksplitting method by ISP. The IntraSubPartitionsSplitType may indicatethe block splitting method by ISP according to ISP information signaledat the coding unit level. If IntraSubPartitionsSplitType is notISP_NO_SPLIT, it may represent that ISP is applied to the current block.Accordingly, if IntraSubPartitionsSplitType is not ISP_NO_SPLIT,transform skip and MTS information may not be signaled and parsed.

Condition v) !cu_sbt_flag: The decoder may check whether sub-blocktransform (SBT) is not applied to the current coding unit. If SBT isapplied to the current coding unit, transform and inverse transform maybe performed according to a preset method. Accordingly, if SBT isapplied to the current coding unit, transform skip and MTS informationmay not be signaled and parsed. The cu_sbt_flag represents a syntaxelement indicating whether SBT is applied to the current coding unit.The cu_sbt_flag may be signaled and parsed at the coding unit level. Ifcu_sbt_flag is 1, it may represent that SBT is applied to the currentcoding unit, and transform skip and MTS information may not be signaledand parsed.

If the conditions i) to v) described above are all true, in step S2902,the decoder may check (or examine) the following the conditions a), b),and c) for determining whether or not to apply the transform skip. As anexample, if the following the conditions a), b), and c) are all true,transform skip-related information may be signaled and parsed. Theconditions a), b), and c) may be defined to be the same as theconditions a), b), and c) previously described in FIG. 27.

In step S2903, if the conditions i) to v) and the conditions a), b), andc) described above are all true, transform_skip_flag[x0][y0], which is asyntax element indicating whether the transform skip is applied to thecurrent luma transform block, may be signaled and parsed. Iftransform_skip_flag[x0][y0] is 1, it may represent that the transform isnot applied to the current luma transform block. Iftransform_skip_flag[x0][y0] is 0, it may represent that transform isapplied to the current luma transform block. Iftransform_skip_flag[x0][y0] does not exist (i.e., is not explicitlysignaled), a value of transform_skip_flag[x0][y0] may be set (orinferred) according to BdpcmFlag[x0][y0]. If BdpcmFlag[x0][y0] is 1, thetransform_skip_flag[x0][y0] may be set to (or inferred as) 1, and ifBdpcmFlag[x0][y0] is 0, the transform_skip_flag[x0][y0] may be set to(or inferred as) 0.

According to an embodiment of the present invention, even if thetransform tree is split into multiple transform units according to themaximum size the condition of the luma transform block (i.e., implicitlysplit on the basis of the luma transform maximum size), if all of theconditions i) to v) and the conditions a), b), and c) are satisfied, thedecoder may parse transform_skip_flag[x0][y0] and also apply transformskip to the split transform units depending on the value oftransform_skip_flag[x0][y0].

When the conditions i) to v) described above are all true, if thefollowing the conditions x), y), and z) are all true (S2904),MTS-related information may be signaled and parsed (S2905).

Condition x) (CuPredMode[chType] [x0] [y0] !=MODE_INTER &&sps_explicit_mts_inter_enabled_flag)∥(CuPredMode[chType][x0][y0]==MODE_INTRA&& sps_explicit_mts_intra_enabled_flag): The decoder may check whetheror not a preset condition is satisfied on the basis of the predictionmode of the current block and whether or not explicit MTS is available.A method of prediction (i.e., a prediction mode) may be determined atthe coding unit level, and a variable CuPredMode[chType][x0][y0] may beset on the basis of information received at the coding unit level. Theprediction mode includes intra prediction and inter prediction.Additionally, the prediction mode may include intra block copy (IBC)prediction for generating a prediction block from a reconstructed regionin the current picture. If CuPred[chType]Mode[x0][y0] is MODE_INTRA, itmay represent that the current block is coded with intra prediction, andif CuPred[chType]Mode[x0][y0] is MODE_INTER, it may represent thecurrent block is coded with inter prediction. In addition, ifCuPred[chType]Mode[x0][y0] is MODE IBC, it may represent that thecurrent block is coded with IBC prediction.

Meanwhile, the flag indicating whether or not MTS is available may besignaled and parsed by being included in at least one of high levelsyntax such as the picture parameter set (PPS), sequence parameter set(SPS), video parameter set (VPS), tile group header, and slice header.Specifically, whether or not MTS is available for the prediction methodmay be independently indicated. If sps_explicit_mts_inter_enabled_flagis 1, it represents that an MTS index (in this specification, it may bereferred to as tu_mts_idx, mts_idx), which is a syntax elementindicating a transform kernel for an inter predicted coding unit, mayexist in the transform unit syntax, and whether or not to apply MTS maybe indicated depending on the MTS index at the transform unit level. Ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex may not exist (i.e., not explicitly signaled) in the transformunit syntax for the inter predicted coding unit, which means that MTSmay not be applied to inter predicted blocks. In other words, ifsps_explicit_mts_inter_enabled_flag is 0, it represents that the MTSindex is not signaled to the inter predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. If MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel. If sps_explicit_mts_intra_enabled_flag is1, it represents that the MTS index may exist in the transform unitsyntax for the intra-predicted coding unit, and whether or not to applyMTS may be indicated depending on the MTS index at the transform unitlevel. If sps_explicit_mts_intra_enabled_flag is 0, it represents thatthe MTS index may not exist in the transform unit syntax (not explicitlysignaled) for the intra-predicted coding unit. In other words, ifsps_explicit_mts_intra_enabled_flag is 0, it represents that the MTSindex is not signaled to the intra-predicted coding unit, and if the MTSindex does not exist, its value may be inferred as 0. If MTS is notapplied, a preset transform kernel may be used, which may be aDCT-II-based transform kernel.

If the current block is coded with intra prediction, and MTS isavailable for intra prediction through high level syntax signaling(i.e., CuPred[chType]Mode[x0][y0] MODE_INTRA &&sps_explicit_mts_intra_enabled_flag) or if the current block is codedwith inter prediction and MTS is available for inter prediction throughhigh level syntax signaling (CuPredMode[chType][x0][y0] !=MODE_INTRA &&sps_explicit_mts_inter_enabled_flag), the condition x) may be true. Thatis, in the condition x), if the corresponding MTS enabled flag is on inthe high level syntax according to the prediction mode (that is,according to the CuPred[chType]Mode[x0][y0] variable), MTS-relatedinformation may be signaled and parsed.

Condition y) !transform_skip_flag[x0][y0]: The decoder may check whethertransform skip is not applied to the current transform block. If thetransform skip is applied to the current transform block, transform isnot applied, and thus MTS-related information may not be signaled andparsed.

Condition z) trafoDepth==0: The decoder may check whether trafoDepth is0. The trafoDepth is a variable representing the degree of splitting (orsplitting depth) of a transform tree. If trafoDepth is 0, it mayrepresent that the size of the coding unit and the size of transformunit are the same. On the other hand, if trafoDepth is greater than 0,it may represent that transform tree splitting is performed by using thecoding unit as a root node, and in this case, the coding unit and thetransform unit may have different sizes. According to an embodiment ofthe present invention, in the transform unit syntax structure of FIG.29, the MTS index may be signaled only if trafoDepth is 0. The decoderdoes not signal an MTS index for a transform unit split according to themaximum size the condition of the luma transform block, and may set theMTS index to a preset value of 0. In this case, the decoder may performtransform using a DCT-II-based basic kernel to the horizontal andvertical directions.

In one embodiment, assuming that the luma maximum transform size is 64and both ISP and SBT are not applied to the current block, if the widthand height of the coding block are both 32 or less, the transform treeis not split, and the trafoDepth variable may always be 0. In this case,the decoder may parse the MTS index, and determine/use a transformkernel according to the MTS index value.

In addition, in one embodiment, assuming that the luma maximum transformsize is 32 and both ISP and SBT are not applied to the current block, ifboth the width and the height of the coding block are 32 or less,whether or not to parse the MTS index may be determined depending ontrafoDepth. If trafoDepth is 0, the transform tree is not split, and thesize of the transform unit may be determined to be the same as the sizeof the coding unit. In this case, the decoder may parse the MTS indexand determine/use a transform kernel according to the MTS index value.On the other hand, if trafoDepth is not 0, it may represent that thetransform tree is split according to the maximum size the condition ofthe luma transform block by using the coding unit as the root node, andthe MTS index is not parsed and may be set to (or inferred as) a presetvalue. The preset value may be 0, and in this case, a DCT-II-based basickernel may be applied to both the horizontal direction and the verticaldirection.

If the conditions i) to v) and the conditions x), y), and z) are alltrue, in step S2905, an MTS index, which is a syntax element indicatingwhich transform kernel is applied to the horizontal and verticaldirections of the residual samples of the luma transform block, may besignaled and parsed. As an embodiment, if the MTS index does not exist,the MTS index may be set to (or inferred as) 0. This may represent thatthe preset basic kernel is applied to both the horizontal direction andthe vertical direction, and the basic kernel may be a DCT-II-basedinteger kernel. The transform kernel applied to the horizontal directionand the vertical direction according to the MTS index may follow themethod previously described in FIG. 14.

FIG. 30 is a flowchart representing a video signal processing methodaccording to an embodiment of the present invention. Referring to FIG.30, a decoder is mainly described for convenience of description, butthe present invention is not limited thereto, and the video signalprocessing method according to the present embodiment may also beapplied to an encoder in substantially the same way.

The decoder checks a prediction mode applied to the current coding unit(S3001).

The decoder checks whether a plurality of preset conditions aresatisfied on the basis of at least one of a prediction mode of thecurrent coding unit or a size of the current coding unit (S3002).

If the plurality of preset conditions are satisfied, the decoder parsesa first syntax element indicating a transform kernel applied to atransform unit included in the current coding unit (S3003).

The decoder determines a transform kernel applied to the horizontal andvertical directions of the current transform unit on the basis of thefirst syntax element (S3004).

The decoder generates a residual signal of the current transform unit byperforming inverse transform on the current transform unit using thedetermined transform kernel (S3005).

As described above, if the plurality of preset conditions are notsatisfied, the first syntax element may be inferred as 0.

In addition, as described above, if the first syntax element is 0, thetransform kernel applied to the horizontal and vertical directions ofthe current transform unit may be determined as a discrete cosinetransform Type-2 (DCT-2)-based transform kernel.

In addition, as described above, in the case of an intra mode, thedecoder may obtain a second syntax element indicating whether a multipletransform selection (MTS) mode may be used in the current sequence.Here, the MTS mode represents a mode in which transform is performedusing a transform kernel selected from among a plurality of transformkernels. In addition, in the case of an inter mode, the decoder mayobtain a third syntax element indicating whether the MTS mode may beused in the current sequence.

In addition, as described above, the plurality of preset conditions mayinclude whether the prediction mode of the current coding unit is anintra mode and the second syntax element is true, or the prediction modeof the current coding unit is an inter mode and the third syntax elementis true.

In addition, as described above, if the prediction mode of the currentcoding unit is an intra block copy mode, the plurality of presetconditions may be determined as being not satisfied regardless of thesecond syntax element and the third syntax element.

In addition, as described above, the plurality of preset conditions mayinclude whether or not a width and height of the current coding unit areless than or equal to 32.

In addition, as described above, a step of receiving information on theluma maximum transform size of the current sequence may be furtherincluded, and if at least one of the width and height of the currentcoding unit is greater than the luma maximum transform size, the currentcoding unit may be split into a plurality of transform units.

In addition, as described above, the luma maximum transform size may be32 or 64.

In addition, as described above, if the current coding unit includes theplurality of transform units split on the basis of the luma maximumtransform size, the plurality of preset conditions may be determined asbeing not satisfied.

The above-described embodiments of the present invention may beimplemented through various means. For example, embodiments of thepresent invention may be implemented by hardware, firmware, software, ora combination thereof.

In the case of implementation by hardware, the method according to theembodiments of the present invention may be implemented by one or moreapplication specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), processors, controllers, microcontrollers, microprocessors, andthe like.

In the case of implementation by firmware or software, the methodaccording to embodiments of the present invention may be implemented inthe form of a module, procedure, or function that performs the functionsor operations described above. The software code may be stored in memoryand driven by a processor. The memory may be located inside or outsidethe processor, and may exchange data with the processor by various meansalready known.

Some embodiments may also be implemented in the form of a recordingmedium including computer-executable instructions such as a programmodule that is executed by a computer. Computer-readable media may beany available media that may be accessed by a computer, and may includeall volatile, nonvolatile, removable, and non-removable media. Inaddition, the computer-readable media may include both computer storagemedia and communication media. The computer storage media include allvolatile, nonvolatile, removable, and non-removable media implemented inany method or technology for storing information such ascomputer-readable instructions, data structures, program modules, orother data. Typically, the communication media include computer-readableinstructions, other data of modulated data signals such as datastructures or program modules, or other transmission mechanisms, andinclude any information transfer media.

The above-mentioned description of the present invention is forillustrative purposes only, and it will be understood that those ofordinary skill in the art to which the present invention belongs maymake changes to the present invention without altering the technicalideas or essential characteristics of the present invention and theinvention may be easily modified in other specific forms. Therefore, theembodiments described above are illustrative and are not restricted inall aspects. For example, each component described as a single entitymay be distributed and implemented, and likewise, components describedas being distributed may also be implemented in an associated fashion.

The scope of the present invention is defined by the appended claimsrather than the above detailed description, and all changes ormodifications derived from the meaning and range of the appended claimsand equivalents thereof are to be interpreted as being included withinthe scope of present invention.

What is claimed is:
 1. A video signal decoding device comprising a processor, wherein the processor is configured to: check a prediction mode applied to a current coding block; check whether a plurality of preset conditions are satisfied on the basis of at least one of a prediction mode of the current coding block or a size of the current coding block; parse when the plurality of preset conditions are satisfied, a first syntax element indicating a transform kernel applied to a transform block included in the current coding block; determine a transform kernel applied to horizontal and vertical directions of a current transform block on the basis of the first syntax element; and generate a residual signal of the current transform block by performing an inverse transform on the current transform block using the determined transform kernel.
 2. The decoding device of claim 1, wherein when the plurality of preset conditions are not satisfied, the first syntax element is inferred as
 0. 3. The decoding device of claim 2, wherein when the first syntax element is 0, the transform kernel applied to the horizontal and vertical directions of the current transform block is determined as a discrete cosine transform type-2 (DCT-2)-based transform kernel.
 4. The decoding device of claim 1, wherein the processor is configured to: obtain, in a case of an intra mode, a second syntax element indicating whether or not a multiple transform selection (MTS) mode is able to be used in a current sequence, the MTS mode representing a mode in which transform is performed using a transform kernel selected from among a plurality of kernels; and obtain, in a case of an inter mode, a third syntax element indicating whether or not the MTS mode is able to be used in the current sequence.
 5. The decoding device of claim 4, wherein the plurality of preset conditions includes whether or not the prediction mode of the current coding block is the intra mode and the second syntax element is true, or the prediction mode of the current coding block is the inter mode and the third syntax element is true.
 6. The decoding device of claim 5, wherein when the prediction mode of the current coding block is an intra block copy mode, the plurality of preset conditions are determined as being not satisfied regardless of the second syntax element and the third syntax element.
 7. The decoding device of claim 1, wherein the plurality of preset conditions includes whether or not a width and height of the current coding block are less than or equal to
 32. 8. The decoding device of claim 1, wherein the processor is configured to: receive information related to a luma maximum transform size of a current sequence, wherein the current coding block, when at least one of the width and height of the current coding block is greater than the luma maximum transform size, is split into a plurality of transform blocks.
 9. The method according to claim 8, wherein the luma maximum transform size is 32 or
 64. 10. The method according to claim 8, wherein when the current coding block includes the plurality of transform blocks split on the basis of the luma maximum transform size, the plurality of preset conditions are determined as being not satisfied.
 11. A non-transitory computer-readable medium storing a bitstream, the bitstream being decoded by a decoding method, wherein the decoding method, comprising: checking a prediction mode applied to a current coding block, checking whether a plurality of preset conditions are satisfied on the basis of at least one of a prediction mode of the current coding block or a size of the current coding block, parsing when the plurality of preset conditions are satisfied, a first syntax element indicating a transform kernel applied to a transform block included in the current coding block, determining a transform kernel applied to horizontal and vertical directions of a current transform block on the basis of the first syntax element, and generating a residual signal of the current transform block by performing an inverse transform on the current transform block using the determined transform kernel.
 12. The non-transitory computer-readable medium of claim 11, wherein when the plurality of preset conditions are not satisfied, the first syntax element is inferred as
 0. 13. The non-transitory computer-readable medium of claim 12, wherein when the first syntax element is 0, the transform kernel applied to the horizontal and vertical directions of the current transform block is determined as a discrete cosine transform type-2 (DCT-2)-based transform kernel.
 14. The non-transitory computer-readable medium 11, the decoding method further comprising: obtaining, in a case of an intra mode, a second syntax element indicating whether or not a multiple transform selection (MTS) mode is able to be used in a current sequence, the MTS mode representing a mode in which transform is performed using a transform kernel selected from among a plurality of kernels, and obtaining, in a case of an inter mode, a third syntax element indicating whether or not the MTS mode is able to be used in the current sequence.
 15. The non-transitory computer-readable medium of claim 14, wherein the plurality of preset conditions includes whether or not the prediction mode of the current coding block is the intra mode and the second syntax element is true, or the prediction mode of the current coding block is the inter mode and the third syntax element is true.
 16. The non-transitory computer-readable medium of claim 15, wherein when the prediction mode of the current coding block is an intra block copy mode, the plurality of preset conditions are determined as being not satisfied regardless of the second syntax element and the third syntax element.
 17. The non-transitory computer-readable medium of claim 11, wherein the plurality of preset conditions includes whether or not a width and height of the current coding block are less than or equal to
 32. 18. The non-transitory computer-readable medium of claim 11, wherein the decoding method further comprising: receiving information related to a luma maximum transform size of a current sequence, and wherein the current coding block, when at least one of the width and height of the current coding block is greater than the luma maximum transform size, is split into a plurality of transform blocks.
 19. The non-transitory computer-readable medium of claim 18, wherein the luma maximum transform size is 32 or
 64. 20. The apparatus of claim 18, wherein when the current coding block includes the plurality of transform blocks split on the basis of the luma maximum transform size, the plurality of preset conditions are determined as being not satisfied.
 21. A video signal encoding device comprising a processor, wherein the processor is configured to: obtain a bitstream to be decoded by a decoder using a decoding method, wherein the decoding method comprising: checking a prediction mode applied to a current coding block, checking whether a plurality of preset conditions are satisfied on the basis of at least one of a prediction mode of the current coding block or a size of the current coding block, parsing when the plurality of preset conditions are satisfied, a first syntax element indicating a transform kernel applied to a transform block included in the current coding block, determining a transform kernel applied to horizontal and vertical directions of a current transform block on the basis of the first syntax element, and generating a residual signal of the current transform block by performing an inverse transform on the current transform block using the determined transform kernel. 